Sep 18 06:22:16.773682 ip-10-0-142-104 systemd[1]: Starting Kubernetes Kubelet... Sep 18 06:22:20.214526 ip-10-0-142-104 kubenswrapper[2567]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 18 06:22:20.214526 ip-10-0-142-104 kubenswrapper[2567]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 18 06:22:20.214526 ip-10-0-142-104 kubenswrapper[2567]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 18 06:22:20.214526 ip-10-0-142-104 kubenswrapper[2567]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 18 06:22:20.214526 ip-10-0-142-104 kubenswrapper[2567]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 18 06:22:20.215314 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.214591 2567 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 18 06:22:20.217478 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217462 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 18 06:22:20.217478 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217477 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 18 06:22:20.217544 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217480 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 18 06:22:20.217544 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217485 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 18 06:22:20.217544 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217488 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 18 06:22:20.217544 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217492 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 18 06:22:20.217544 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217495 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 18 06:22:20.217544 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217498 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 18 06:22:20.217544 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217502 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 18 06:22:20.217544 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217506 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 18 06:22:20.217544 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217510 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 18 06:22:20.217544 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217513 2567 feature_gate.go:328] unrecognized feature gate: Example2 Sep 18 06:22:20.217544 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217516 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 18 06:22:20.217544 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217520 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 18 06:22:20.217544 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217523 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 18 06:22:20.217544 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217526 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 18 06:22:20.217544 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217529 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 18 06:22:20.217544 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217534 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 18 06:22:20.217544 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217537 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 18 06:22:20.217544 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217547 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 18 06:22:20.217544 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217551 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 18 06:22:20.218005 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217555 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 18 06:22:20.218005 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217558 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 18 06:22:20.218005 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217561 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 18 06:22:20.218005 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217564 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 18 06:22:20.218005 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217567 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 18 06:22:20.218005 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217570 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 18 06:22:20.218005 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217572 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 18 06:22:20.218005 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217575 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 18 06:22:20.218005 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217578 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 18 06:22:20.218005 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217581 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 18 06:22:20.218005 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217584 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 18 06:22:20.218005 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217587 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 18 06:22:20.218005 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217590 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 18 06:22:20.218005 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217592 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 18 06:22:20.218005 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217595 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 18 06:22:20.218005 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217598 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 18 06:22:20.218005 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217601 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 18 06:22:20.218005 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217603 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 18 06:22:20.218005 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217606 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 18 06:22:20.218005 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217609 2567 feature_gate.go:328] unrecognized feature gate: Example Sep 18 06:22:20.218502 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217612 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 18 06:22:20.218502 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217614 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 18 06:22:20.218502 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217617 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 18 06:22:20.218502 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217620 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 18 06:22:20.218502 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217622 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 18 06:22:20.218502 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217626 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 18 06:22:20.218502 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217628 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 18 06:22:20.218502 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217631 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 18 06:22:20.218502 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217633 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 18 06:22:20.218502 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217636 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 18 06:22:20.218502 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217639 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 18 06:22:20.218502 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217641 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 18 06:22:20.218502 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217644 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 18 06:22:20.218502 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217647 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 18 06:22:20.218502 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217650 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 18 06:22:20.218502 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217653 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 18 06:22:20.218502 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217655 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 18 06:22:20.218502 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217658 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 18 06:22:20.218502 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217662 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 18 06:22:20.218502 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217666 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 18 06:22:20.218991 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217669 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 18 06:22:20.218991 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217672 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 18 06:22:20.218991 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217674 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 18 06:22:20.218991 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217677 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 18 06:22:20.218991 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217680 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 18 06:22:20.218991 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217682 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 18 06:22:20.218991 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217685 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 18 06:22:20.218991 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217688 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 18 06:22:20.218991 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217690 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 18 06:22:20.218991 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217693 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 18 06:22:20.218991 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217696 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 18 06:22:20.218991 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217698 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 18 06:22:20.218991 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217701 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 18 06:22:20.218991 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217703 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 18 06:22:20.218991 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217706 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 18 06:22:20.218991 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217709 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 18 06:22:20.218991 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217711 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 18 06:22:20.218991 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217714 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 18 06:22:20.218991 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217717 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 18 06:22:20.218991 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217720 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 18 06:22:20.219484 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217722 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 18 06:22:20.219484 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217725 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 18 06:22:20.219484 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217728 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 18 06:22:20.219484 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217730 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 18 06:22:20.219484 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217733 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 18 06:22:20.219484 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217738 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 18 06:22:20.219484 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217741 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 18 06:22:20.219484 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217743 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 18 06:22:20.219484 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217746 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 18 06:22:20.219484 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217749 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 18 06:22:20.219484 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217752 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 18 06:22:20.219484 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217755 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 18 06:22:20.219484 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217757 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 18 06:22:20.219484 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217760 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 18 06:22:20.219484 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217763 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 18 06:22:20.219484 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217765 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 18 06:22:20.219484 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217768 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 18 06:22:20.219484 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.217771 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 18 06:22:20.219484 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218450 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 18 06:22:20.219484 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218456 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 18 06:22:20.219969 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218460 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 18 06:22:20.219969 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218465 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 18 06:22:20.219969 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218469 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 18 06:22:20.219969 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218472 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 18 06:22:20.219969 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218476 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 18 06:22:20.219969 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218478 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 18 06:22:20.219969 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218483 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 18 06:22:20.219969 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218486 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 18 06:22:20.219969 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218488 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 18 06:22:20.219969 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218491 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 18 06:22:20.219969 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218494 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 18 06:22:20.219969 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218498 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 18 06:22:20.219969 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218501 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 18 06:22:20.219969 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218504 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 18 06:22:20.219969 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218506 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 18 06:22:20.219969 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218509 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 18 06:22:20.219969 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218512 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 18 06:22:20.219969 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218516 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 18 06:22:20.219969 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218519 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 18 06:22:20.219969 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218521 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 18 06:22:20.220528 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218524 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 18 06:22:20.220528 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218527 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 18 06:22:20.220528 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218530 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 18 06:22:20.220528 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218532 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 18 06:22:20.220528 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218535 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 18 06:22:20.220528 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218538 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 18 06:22:20.220528 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218540 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 18 06:22:20.220528 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218543 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 18 06:22:20.220528 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218546 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 18 06:22:20.220528 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218549 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 18 06:22:20.220528 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218551 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 18 06:22:20.220528 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218554 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 18 06:22:20.220528 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218558 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 18 06:22:20.220528 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218562 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 18 06:22:20.220528 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218565 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 18 06:22:20.220528 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218569 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 18 06:22:20.220528 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218572 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 18 06:22:20.220528 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218575 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 18 06:22:20.220528 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218578 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 18 06:22:20.220528 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218580 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 18 06:22:20.221015 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218583 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 18 06:22:20.221015 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218586 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 18 06:22:20.221015 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218588 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 18 06:22:20.221015 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218591 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 18 06:22:20.221015 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218599 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 18 06:22:20.221015 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218602 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 18 06:22:20.221015 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218606 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 18 06:22:20.221015 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218608 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 18 06:22:20.221015 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218611 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 18 06:22:20.221015 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218615 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 18 06:22:20.221015 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218618 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 18 06:22:20.221015 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218621 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 18 06:22:20.221015 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218624 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 18 06:22:20.221015 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218627 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 18 06:22:20.221015 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218629 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 18 06:22:20.221015 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218632 2567 feature_gate.go:328] unrecognized feature gate: Example2 Sep 18 06:22:20.221015 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218634 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 18 06:22:20.221015 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218639 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 18 06:22:20.221015 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218641 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 18 06:22:20.221506 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218644 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 18 06:22:20.221506 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218647 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 18 06:22:20.221506 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218649 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 18 06:22:20.221506 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218652 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 18 06:22:20.221506 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218655 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 18 06:22:20.221506 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218658 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 18 06:22:20.221506 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218660 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 18 06:22:20.221506 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218663 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 18 06:22:20.221506 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218665 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 18 06:22:20.221506 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218668 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 18 06:22:20.221506 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218671 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 18 06:22:20.221506 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218673 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 18 06:22:20.221506 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218676 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 18 06:22:20.221506 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218679 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 18 06:22:20.221506 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218681 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 18 06:22:20.221506 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218684 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 18 06:22:20.221506 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218687 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 18 06:22:20.221506 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218689 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 18 06:22:20.221506 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218692 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 18 06:22:20.221506 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218695 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 18 06:22:20.221997 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218698 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 18 06:22:20.221997 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218701 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 18 06:22:20.221997 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218704 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 18 06:22:20.221997 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218707 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 18 06:22:20.221997 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218710 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 18 06:22:20.221997 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218713 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 18 06:22:20.221997 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218716 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 18 06:22:20.221997 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218718 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 18 06:22:20.221997 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218721 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 18 06:22:20.221997 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218723 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 18 06:22:20.221997 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218726 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 18 06:22:20.221997 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218729 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 18 06:22:20.221997 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218731 2567 feature_gate.go:328] unrecognized feature gate: Example Sep 18 06:22:20.221997 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218734 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 18 06:22:20.221997 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218736 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 18 06:22:20.221997 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218739 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 18 06:22:20.221997 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218742 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 18 06:22:20.221997 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.218745 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 18 06:22:20.221997 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219142 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 18 06:22:20.221997 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219148 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 18 06:22:20.222500 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219151 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 18 06:22:20.222500 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219154 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 18 06:22:20.222500 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219157 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 18 06:22:20.222500 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219160 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 18 06:22:20.222500 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219163 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 18 06:22:20.222500 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219166 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 18 06:22:20.222500 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219169 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 18 06:22:20.222500 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219172 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 18 06:22:20.222500 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219175 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 18 06:22:20.222500 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219178 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 18 06:22:20.222500 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219186 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 18 06:22:20.222500 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219189 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 18 06:22:20.222500 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219192 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 18 06:22:20.222500 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219195 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 18 06:22:20.222500 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219198 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 18 06:22:20.222500 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219201 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 18 06:22:20.222500 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219204 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 18 06:22:20.222500 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219207 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 18 06:22:20.222500 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219210 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 18 06:22:20.222500 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219213 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 18 06:22:20.222997 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219216 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 18 06:22:20.222997 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219218 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 18 06:22:20.222997 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219221 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 18 06:22:20.222997 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219224 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 18 06:22:20.222997 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219227 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 18 06:22:20.222997 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219230 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 18 06:22:20.222997 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219232 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 18 06:22:20.222997 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219235 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 18 06:22:20.222997 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219237 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 18 06:22:20.222997 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219240 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 18 06:22:20.222997 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219243 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 18 06:22:20.222997 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219246 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 18 06:22:20.222997 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219250 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 18 06:22:20.222997 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219254 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 18 06:22:20.222997 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219257 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 18 06:22:20.222997 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219260 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 18 06:22:20.222997 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219263 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 18 06:22:20.222997 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219266 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 18 06:22:20.222997 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219269 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 18 06:22:20.223470 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219271 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 18 06:22:20.223470 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219274 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 18 06:22:20.223470 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219276 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 18 06:22:20.223470 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219279 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 18 06:22:20.223470 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219283 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 18 06:22:20.223470 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219286 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 18 06:22:20.223470 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219289 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 18 06:22:20.223470 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219291 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 18 06:22:20.223470 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219294 2567 feature_gate.go:328] unrecognized feature gate: Example Sep 18 06:22:20.223470 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219297 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 18 06:22:20.223470 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219300 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 18 06:22:20.223470 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219302 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 18 06:22:20.223470 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219305 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 18 06:22:20.223470 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219308 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 18 06:22:20.223470 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219310 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 18 06:22:20.223470 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219313 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 18 06:22:20.223470 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219316 2567 feature_gate.go:328] unrecognized feature gate: Example2 Sep 18 06:22:20.223470 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219319 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 18 06:22:20.223470 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219322 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 18 06:22:20.223470 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219325 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 18 06:22:20.223947 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219327 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 18 06:22:20.223947 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219333 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 18 06:22:20.223947 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219336 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 18 06:22:20.223947 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219339 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 18 06:22:20.223947 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219342 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 18 06:22:20.223947 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219344 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 18 06:22:20.223947 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219347 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 18 06:22:20.223947 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219349 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 18 06:22:20.223947 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219352 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 18 06:22:20.223947 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219355 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 18 06:22:20.223947 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219357 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 18 06:22:20.223947 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219360 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 18 06:22:20.223947 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219380 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 18 06:22:20.223947 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219384 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 18 06:22:20.223947 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219391 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 18 06:22:20.223947 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219395 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 18 06:22:20.223947 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219398 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 18 06:22:20.223947 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219401 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 18 06:22:20.223947 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219404 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 18 06:22:20.223947 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219407 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 18 06:22:20.224434 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219409 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 18 06:22:20.224434 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219412 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 18 06:22:20.224434 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219415 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 18 06:22:20.224434 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219417 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 18 06:22:20.224434 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219420 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 18 06:22:20.224434 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219422 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 18 06:22:20.224434 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219427 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 18 06:22:20.224434 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219430 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 18 06:22:20.224434 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219432 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 18 06:22:20.224434 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219435 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 18 06:22:20.224434 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219437 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 18 06:22:20.224434 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219440 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 18 06:22:20.224434 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219443 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 18 06:22:20.224434 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219446 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 18 06:22:20.224434 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219448 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 18 06:22:20.224434 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219452 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 18 06:22:20.224434 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219454 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 18 06:22:20.224434 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.219459 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 18 06:22:20.224434 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219519 2567 flags.go:64] FLAG: --address="0.0.0.0" Sep 18 06:22:20.224434 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219552 2567 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 18 06:22:20.224910 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219561 2567 flags.go:64] FLAG: --anonymous-auth="true" Sep 18 06:22:20.224910 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219566 2567 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 18 06:22:20.224910 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219570 2567 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 18 06:22:20.224910 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219573 2567 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 18 06:22:20.224910 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219578 2567 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 18 06:22:20.224910 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219582 2567 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 18 06:22:20.224910 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219585 2567 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 18 06:22:20.224910 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219591 2567 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 18 06:22:20.224910 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219595 2567 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 18 06:22:20.224910 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219598 2567 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 18 06:22:20.224910 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219602 2567 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 18 06:22:20.224910 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219606 2567 flags.go:64] FLAG: --cgroup-root="" Sep 18 06:22:20.224910 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219609 2567 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 18 06:22:20.224910 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219612 2567 flags.go:64] FLAG: --client-ca-file="" Sep 18 06:22:20.224910 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219616 2567 flags.go:64] FLAG: --cloud-provider="external" Sep 18 06:22:20.224910 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219619 2567 flags.go:64] FLAG: --cluster-dns="[]" Sep 18 06:22:20.224910 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219624 2567 flags.go:64] FLAG: --cluster-domain="" Sep 18 06:22:20.224910 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219627 2567 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 18 06:22:20.224910 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219630 2567 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 18 06:22:20.224910 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219634 2567 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 18 06:22:20.224910 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219638 2567 flags.go:64] FLAG: --container-log-max-files="5" Sep 18 06:22:20.224910 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219642 2567 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 18 06:22:20.224910 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219645 2567 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 18 06:22:20.225530 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219648 2567 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 18 06:22:20.225530 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219652 2567 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 18 06:22:20.225530 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219655 2567 flags.go:64] FLAG: --contention-profiling="false" Sep 18 06:22:20.225530 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219659 2567 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 18 06:22:20.225530 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219662 2567 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 18 06:22:20.225530 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219666 2567 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 18 06:22:20.225530 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219670 2567 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 18 06:22:20.225530 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219674 2567 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 18 06:22:20.225530 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219677 2567 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 18 06:22:20.225530 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219681 2567 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 18 06:22:20.225530 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219684 2567 flags.go:64] FLAG: --enable-load-reader="false" Sep 18 06:22:20.225530 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219688 2567 flags.go:64] FLAG: --enable-server="true" Sep 18 06:22:20.225530 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219691 2567 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 18 06:22:20.225530 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219695 2567 flags.go:64] FLAG: --event-burst="100" Sep 18 06:22:20.225530 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219698 2567 flags.go:64] FLAG: --event-qps="50" Sep 18 06:22:20.225530 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219702 2567 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 18 06:22:20.225530 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219706 2567 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 18 06:22:20.225530 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219710 2567 flags.go:64] FLAG: --eviction-hard="" Sep 18 06:22:20.225530 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219713 2567 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 18 06:22:20.225530 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219716 2567 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 18 06:22:20.225530 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219720 2567 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 18 06:22:20.225530 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219724 2567 flags.go:64] FLAG: --eviction-soft="" Sep 18 06:22:20.225530 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219727 2567 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 18 06:22:20.225530 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219730 2567 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 18 06:22:20.225530 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219733 2567 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 18 06:22:20.226145 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219736 2567 flags.go:64] FLAG: --experimental-mounter-path="" Sep 18 06:22:20.226145 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219740 2567 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 18 06:22:20.226145 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219743 2567 flags.go:64] FLAG: --fail-swap-on="true" Sep 18 06:22:20.226145 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219746 2567 flags.go:64] FLAG: --feature-gates="" Sep 18 06:22:20.226145 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219750 2567 flags.go:64] FLAG: --file-check-frequency="20s" Sep 18 06:22:20.226145 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219753 2567 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 18 06:22:20.226145 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219756 2567 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 18 06:22:20.226145 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219760 2567 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 18 06:22:20.226145 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219763 2567 flags.go:64] FLAG: --healthz-port="10248" Sep 18 06:22:20.226145 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219767 2567 flags.go:64] FLAG: --help="false" Sep 18 06:22:20.226145 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219770 2567 flags.go:64] FLAG: --hostname-override="ip-10-0-142-104.ec2.internal" Sep 18 06:22:20.226145 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219774 2567 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 18 06:22:20.226145 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219778 2567 flags.go:64] FLAG: --http-check-frequency="20s" Sep 18 06:22:20.226145 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219781 2567 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 18 06:22:20.226145 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219785 2567 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 18 06:22:20.226145 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219788 2567 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 18 06:22:20.226145 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219792 2567 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 18 06:22:20.226145 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219795 2567 flags.go:64] FLAG: --image-service-endpoint="" Sep 18 06:22:20.226145 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219798 2567 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 18 06:22:20.226145 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219801 2567 flags.go:64] FLAG: --kube-api-burst="100" Sep 18 06:22:20.226145 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219804 2567 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 18 06:22:20.226145 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219807 2567 flags.go:64] FLAG: --kube-api-qps="50" Sep 18 06:22:20.226145 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219810 2567 flags.go:64] FLAG: --kube-reserved="" Sep 18 06:22:20.226145 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219816 2567 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 18 06:22:20.226782 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219819 2567 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 18 06:22:20.226782 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219822 2567 flags.go:64] FLAG: --kubelet-cgroups="" Sep 18 06:22:20.226782 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219825 2567 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 18 06:22:20.226782 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219828 2567 flags.go:64] FLAG: --lock-file="" Sep 18 06:22:20.226782 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219832 2567 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 18 06:22:20.226782 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219835 2567 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 18 06:22:20.226782 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219838 2567 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 18 06:22:20.226782 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219844 2567 flags.go:64] FLAG: --log-json-split-stream="false" Sep 18 06:22:20.226782 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219847 2567 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 18 06:22:20.226782 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219850 2567 flags.go:64] FLAG: --log-text-split-stream="false" Sep 18 06:22:20.226782 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219853 2567 flags.go:64] FLAG: --logging-format="text" Sep 18 06:22:20.226782 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219856 2567 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 18 06:22:20.226782 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219860 2567 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 18 06:22:20.226782 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219863 2567 flags.go:64] FLAG: --manifest-url="" Sep 18 06:22:20.226782 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219866 2567 flags.go:64] FLAG: --manifest-url-header="" Sep 18 06:22:20.226782 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219870 2567 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 18 06:22:20.226782 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219873 2567 flags.go:64] FLAG: --max-open-files="1000000" Sep 18 06:22:20.226782 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219877 2567 flags.go:64] FLAG: --max-pods="110" Sep 18 06:22:20.226782 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219881 2567 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 18 06:22:20.226782 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219886 2567 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 18 06:22:20.226782 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219889 2567 flags.go:64] FLAG: --memory-manager-policy="None" Sep 18 06:22:20.226782 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219892 2567 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 18 06:22:20.226782 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219895 2567 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 18 06:22:20.226782 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219899 2567 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 18 06:22:20.226782 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219902 2567 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 18 06:22:20.227847 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219909 2567 flags.go:64] FLAG: --node-status-max-images="50" Sep 18 06:22:20.227847 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219912 2567 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 18 06:22:20.227847 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219915 2567 flags.go:64] FLAG: --oom-score-adj="-999" Sep 18 06:22:20.227847 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219919 2567 flags.go:64] FLAG: --pod-cidr="" Sep 18 06:22:20.227847 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219922 2567 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc" Sep 18 06:22:20.227847 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219927 2567 flags.go:64] FLAG: --pod-manifest-path="" Sep 18 06:22:20.227847 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219930 2567 flags.go:64] FLAG: --pod-max-pids="-1" Sep 18 06:22:20.227847 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219935 2567 flags.go:64] FLAG: --pods-per-core="0" Sep 18 06:22:20.227847 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219938 2567 flags.go:64] FLAG: --port="10250" Sep 18 06:22:20.227847 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219942 2567 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 18 06:22:20.227847 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219945 2567 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06ba5fb6972484e05" Sep 18 06:22:20.227847 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219949 2567 flags.go:64] FLAG: --qos-reserved="" Sep 18 06:22:20.227847 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219953 2567 flags.go:64] FLAG: --read-only-port="10255" Sep 18 06:22:20.227847 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219957 2567 flags.go:64] FLAG: --register-node="true" Sep 18 06:22:20.227847 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219960 2567 flags.go:64] FLAG: --register-with-taints="" Sep 18 06:22:20.227847 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219964 2567 flags.go:64] FLAG: --registry-burst="10" Sep 18 06:22:20.227847 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219967 2567 flags.go:64] FLAG: --registry-qps="5" Sep 18 06:22:20.227847 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219970 2567 flags.go:64] FLAG: --reserved-cpus="" Sep 18 06:22:20.227847 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219973 2567 flags.go:64] FLAG: --reserved-memory="" Sep 18 06:22:20.227847 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219978 2567 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 18 06:22:20.227847 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219982 2567 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 18 06:22:20.227847 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219985 2567 flags.go:64] FLAG: --rotate-certificates="false" Sep 18 06:22:20.227847 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219988 2567 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 18 06:22:20.227847 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219991 2567 flags.go:64] FLAG: --runonce="false" Sep 18 06:22:20.227847 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219994 2567 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 18 06:22:20.228901 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.219998 2567 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 18 06:22:20.228901 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.220003 2567 flags.go:64] FLAG: --seccomp-default="false" Sep 18 06:22:20.228901 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.220006 2567 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 18 06:22:20.228901 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.220009 2567 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 18 06:22:20.228901 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.220012 2567 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 18 06:22:20.228901 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.220016 2567 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 18 06:22:20.228901 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.220019 2567 flags.go:64] FLAG: --storage-driver-password="root" Sep 18 06:22:20.228901 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.220022 2567 flags.go:64] FLAG: --storage-driver-secure="false" Sep 18 06:22:20.228901 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.220025 2567 flags.go:64] FLAG: --storage-driver-table="stats" Sep 18 06:22:20.228901 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.220028 2567 flags.go:64] FLAG: --storage-driver-user="root" Sep 18 06:22:20.228901 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.220032 2567 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 18 06:22:20.228901 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.220035 2567 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 18 06:22:20.228901 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.220038 2567 flags.go:64] FLAG: --system-cgroups="" Sep 18 06:22:20.228901 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.220041 2567 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 18 06:22:20.228901 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.220048 2567 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 18 06:22:20.228901 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.220051 2567 flags.go:64] FLAG: --tls-cert-file="" Sep 18 06:22:20.228901 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.220054 2567 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 18 06:22:20.228901 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.220059 2567 flags.go:64] FLAG: --tls-min-version="" Sep 18 06:22:20.228901 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.220062 2567 flags.go:64] FLAG: --tls-private-key-file="" Sep 18 06:22:20.228901 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.220065 2567 flags.go:64] FLAG: --topology-manager-policy="none" Sep 18 06:22:20.228901 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.220069 2567 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 18 06:22:20.228901 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.220072 2567 flags.go:64] FLAG: --topology-manager-scope="container" Sep 18 06:22:20.228901 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.220076 2567 flags.go:64] FLAG: --v="2" Sep 18 06:22:20.228901 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.220079 2567 flags.go:64] FLAG: --version="false" Sep 18 06:22:20.228901 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.220083 2567 flags.go:64] FLAG: --vmodule="" Sep 18 06:22:20.229945 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.220087 2567 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 18 06:22:20.229945 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.220091 2567 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 18 06:22:20.229945 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220200 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 18 06:22:20.229945 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220203 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 18 06:22:20.229945 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220206 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 18 06:22:20.229945 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220209 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 18 06:22:20.229945 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220212 2567 feature_gate.go:328] unrecognized feature gate: Example2 Sep 18 06:22:20.229945 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220215 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 18 06:22:20.229945 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220219 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 18 06:22:20.229945 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220222 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 18 06:22:20.229945 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220225 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 18 06:22:20.229945 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220228 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 18 06:22:20.229945 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220230 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 18 06:22:20.229945 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220233 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 18 06:22:20.229945 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220236 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 18 06:22:20.229945 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220239 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 18 06:22:20.229945 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220241 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 18 06:22:20.229945 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220244 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 18 06:22:20.229945 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220247 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 18 06:22:20.229945 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220249 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 18 06:22:20.230853 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220252 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 18 06:22:20.230853 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220256 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 18 06:22:20.230853 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220259 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 18 06:22:20.230853 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220262 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 18 06:22:20.230853 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220265 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 18 06:22:20.230853 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220267 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 18 06:22:20.230853 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220270 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 18 06:22:20.230853 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220273 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 18 06:22:20.230853 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220276 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 18 06:22:20.230853 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220279 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 18 06:22:20.230853 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220281 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 18 06:22:20.230853 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220284 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 18 06:22:20.230853 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220287 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 18 06:22:20.230853 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220290 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 18 06:22:20.230853 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220293 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 18 06:22:20.230853 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220295 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 18 06:22:20.230853 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220298 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 18 06:22:20.230853 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220301 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 18 06:22:20.230853 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220304 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 18 06:22:20.231727 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220306 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 18 06:22:20.231727 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220311 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 18 06:22:20.231727 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220314 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 18 06:22:20.231727 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220317 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 18 06:22:20.231727 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220320 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 18 06:22:20.231727 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220322 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 18 06:22:20.231727 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220325 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 18 06:22:20.231727 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220328 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 18 06:22:20.231727 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220331 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 18 06:22:20.231727 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220334 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 18 06:22:20.231727 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220336 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 18 06:22:20.231727 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220339 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 18 06:22:20.231727 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220342 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 18 06:22:20.231727 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220344 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 18 06:22:20.231727 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220348 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 18 06:22:20.231727 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220351 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 18 06:22:20.231727 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220354 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 18 06:22:20.231727 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220358 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 18 06:22:20.231727 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220376 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 18 06:22:20.232321 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220380 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 18 06:22:20.232321 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220383 2567 feature_gate.go:328] unrecognized feature gate: Example Sep 18 06:22:20.232321 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220386 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 18 06:22:20.232321 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220389 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 18 06:22:20.232321 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220392 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 18 06:22:20.232321 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220395 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 18 06:22:20.232321 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220398 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 18 06:22:20.232321 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220401 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 18 06:22:20.232321 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220404 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 18 06:22:20.232321 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220407 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 18 06:22:20.232321 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220410 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 18 06:22:20.232321 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220414 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 18 06:22:20.232321 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220418 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 18 06:22:20.232321 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220421 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 18 06:22:20.232321 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220425 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 18 06:22:20.232321 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220428 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 18 06:22:20.232321 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220430 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 18 06:22:20.232321 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220433 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 18 06:22:20.232321 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220436 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 18 06:22:20.232321 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220439 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 18 06:22:20.233000 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220441 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 18 06:22:20.233000 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220444 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 18 06:22:20.233000 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220447 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 18 06:22:20.233000 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220450 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 18 06:22:20.233000 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220453 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 18 06:22:20.233000 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220455 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 18 06:22:20.233000 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220458 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 18 06:22:20.233000 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220462 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 18 06:22:20.233000 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220465 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 18 06:22:20.233000 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220468 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 18 06:22:20.233000 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220472 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 18 06:22:20.233000 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220475 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 18 06:22:20.233000 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220479 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 18 06:22:20.233000 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220482 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 18 06:22:20.233000 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220486 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 18 06:22:20.233000 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220489 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 18 06:22:20.233000 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220492 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 18 06:22:20.233000 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220495 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 18 06:22:20.233000 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220498 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 18 06:22:20.233810 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220501 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 18 06:22:20.233810 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220504 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 18 06:22:20.233810 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220507 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 18 06:22:20.233810 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.220510 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 18 06:22:20.233810 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.221031 2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 18 06:22:20.233810 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.230500 2567 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 18 06:22:20.233810 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.230521 2567 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 18 06:22:20.233810 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.230547 2567 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 18 06:22:20.233810 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.230556 2567 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 18 06:22:20.233810 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230639 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 18 06:22:20.233810 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230646 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 18 06:22:20.233810 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230651 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 18 06:22:20.233810 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230656 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 18 06:22:20.233810 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230661 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 18 06:22:20.233810 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230666 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 18 06:22:20.234360 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230671 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 18 06:22:20.234360 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230676 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 18 06:22:20.234360 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230681 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 18 06:22:20.234360 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230685 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 18 06:22:20.234360 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230689 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 18 06:22:20.234360 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230694 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 18 06:22:20.234360 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230698 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 18 06:22:20.234360 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230703 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 18 06:22:20.234360 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230707 2567 feature_gate.go:328] unrecognized feature gate: Example Sep 18 06:22:20.234360 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230711 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 18 06:22:20.234360 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230716 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 18 06:22:20.234360 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230720 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 18 06:22:20.234360 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230725 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 18 06:22:20.234360 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230729 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 18 06:22:20.234360 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230737 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 18 06:22:20.234360 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230744 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 18 06:22:20.234360 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230750 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 18 06:22:20.234360 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230754 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 18 06:22:20.234360 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230759 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 18 06:22:20.234360 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230764 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 18 06:22:20.234999 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230768 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 18 06:22:20.234999 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230773 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 18 06:22:20.234999 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230778 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 18 06:22:20.234999 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230784 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 18 06:22:20.234999 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230789 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 18 06:22:20.234999 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230793 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 18 06:22:20.234999 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230797 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 18 06:22:20.234999 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230802 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 18 06:22:20.234999 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230806 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 18 06:22:20.234999 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230810 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 18 06:22:20.234999 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230815 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 18 06:22:20.234999 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230820 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 18 06:22:20.234999 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230824 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 18 06:22:20.234999 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230829 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 18 06:22:20.234999 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230833 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 18 06:22:20.234999 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230838 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 18 06:22:20.234999 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230842 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 18 06:22:20.234999 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230846 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 18 06:22:20.234999 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230852 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 18 06:22:20.235579 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230858 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 18 06:22:20.235579 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230862 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 18 06:22:20.235579 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230867 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 18 06:22:20.235579 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230871 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 18 06:22:20.235579 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230876 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 18 06:22:20.235579 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230880 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 18 06:22:20.235579 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230885 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 18 06:22:20.235579 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230891 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 18 06:22:20.235579 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230896 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 18 06:22:20.235579 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230900 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 18 06:22:20.235579 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230905 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 18 06:22:20.235579 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230909 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 18 06:22:20.235579 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230913 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 18 06:22:20.235579 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230918 2567 feature_gate.go:328] unrecognized feature gate: Example2 Sep 18 06:22:20.235579 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230922 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 18 06:22:20.235579 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230927 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 18 06:22:20.235579 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230934 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 18 06:22:20.235579 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230940 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 18 06:22:20.235579 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230945 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 18 06:22:20.236107 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230949 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 18 06:22:20.236107 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230953 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 18 06:22:20.236107 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230957 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 18 06:22:20.236107 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230962 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 18 06:22:20.236107 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230966 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 18 06:22:20.236107 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230970 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 18 06:22:20.236107 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230975 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 18 06:22:20.236107 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230980 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 18 06:22:20.236107 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230984 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 18 06:22:20.236107 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230989 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 18 06:22:20.236107 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230993 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 18 06:22:20.236107 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.230997 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 18 06:22:20.236107 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231002 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 18 06:22:20.236107 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231006 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 18 06:22:20.236107 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231010 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 18 06:22:20.236107 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231016 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 18 06:22:20.236107 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231021 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 18 06:22:20.236107 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231026 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 18 06:22:20.236107 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231030 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 18 06:22:20.236107 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231034 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 18 06:22:20.236711 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231039 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 18 06:22:20.236711 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231043 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 18 06:22:20.236711 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231048 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 18 06:22:20.236711 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231052 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 18 06:22:20.236711 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231056 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 18 06:22:20.236711 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231061 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 18 06:22:20.236711 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231065 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 18 06:22:20.236711 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231069 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 18 06:22:20.236711 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231074 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 18 06:22:20.236711 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231079 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 18 06:22:20.236711 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231083 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 18 06:22:20.236711 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231088 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 18 06:22:20.236711 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231092 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 18 06:22:20.236711 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231097 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 18 06:22:20.236711 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231101 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 18 06:22:20.237082 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.231109 2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 18 06:22:20.237082 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231285 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 18 06:22:20.237082 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231294 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 18 06:22:20.237082 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231298 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 18 06:22:20.237082 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231303 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 18 06:22:20.237082 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231307 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 18 06:22:20.237082 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231311 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 18 06:22:20.237082 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231316 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 18 06:22:20.237082 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231320 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 18 06:22:20.237082 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231325 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 18 06:22:20.237082 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231330 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 18 06:22:20.237082 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231334 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 18 06:22:20.237082 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231340 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 18 06:22:20.237082 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231348 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 18 06:22:20.237487 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231353 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 18 06:22:20.237487 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231358 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 18 06:22:20.237487 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231380 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 18 06:22:20.237487 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231385 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 18 06:22:20.237487 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231390 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 18 06:22:20.237487 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231395 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 18 06:22:20.237487 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231399 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 18 06:22:20.237487 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231403 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 18 06:22:20.237487 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231408 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 18 06:22:20.237487 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231412 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 18 06:22:20.237487 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231416 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 18 06:22:20.237487 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231421 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 18 06:22:20.237487 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231425 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 18 06:22:20.237487 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231430 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 18 06:22:20.237487 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231434 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 18 06:22:20.237487 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231438 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 18 06:22:20.237487 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231443 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 18 06:22:20.237487 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231447 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 18 06:22:20.237487 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231451 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 18 06:22:20.237487 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231455 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 18 06:22:20.237990 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231460 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 18 06:22:20.237990 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231464 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 18 06:22:20.237990 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231468 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 18 06:22:20.237990 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231472 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 18 06:22:20.237990 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231476 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 18 06:22:20.237990 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231481 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 18 06:22:20.237990 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231485 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 18 06:22:20.237990 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231489 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 18 06:22:20.237990 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231496 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 18 06:22:20.237990 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231501 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 18 06:22:20.237990 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231506 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 18 06:22:20.237990 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231511 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 18 06:22:20.237990 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231517 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 18 06:22:20.237990 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231522 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 18 06:22:20.237990 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231526 2567 feature_gate.go:328] unrecognized feature gate: Example2 Sep 18 06:22:20.237990 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231530 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 18 06:22:20.237990 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231534 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 18 06:22:20.237990 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231539 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 18 06:22:20.237990 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231543 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 18 06:22:20.238790 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231547 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 18 06:22:20.238790 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231551 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 18 06:22:20.238790 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231556 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 18 06:22:20.238790 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231560 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 18 06:22:20.238790 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231564 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 18 06:22:20.238790 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231569 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 18 06:22:20.238790 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231575 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 18 06:22:20.238790 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231581 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 18 06:22:20.238790 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231585 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 18 06:22:20.238790 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231590 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 18 06:22:20.238790 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231595 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 18 06:22:20.238790 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231599 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 18 06:22:20.238790 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231604 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 18 06:22:20.238790 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231608 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 18 06:22:20.238790 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231612 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 18 06:22:20.238790 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231616 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 18 06:22:20.238790 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231621 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 18 06:22:20.238790 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231625 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 18 06:22:20.238790 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231629 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 18 06:22:20.238790 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231633 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 18 06:22:20.239529 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231637 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 18 06:22:20.239529 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231641 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 18 06:22:20.239529 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231645 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 18 06:22:20.239529 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231650 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 18 06:22:20.239529 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231654 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 18 06:22:20.239529 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231659 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 18 06:22:20.239529 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231664 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 18 06:22:20.239529 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231669 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 18 06:22:20.239529 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231673 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 18 06:22:20.239529 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231677 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 18 06:22:20.239529 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231681 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 18 06:22:20.239529 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231686 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 18 06:22:20.239529 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231690 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 18 06:22:20.239529 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231694 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 18 06:22:20.239529 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231698 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 18 06:22:20.239529 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231702 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 18 06:22:20.239529 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231706 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 18 06:22:20.239529 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231711 2567 feature_gate.go:328] unrecognized feature gate: Example Sep 18 06:22:20.239529 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231715 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 18 06:22:20.239529 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231719 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 18 06:22:20.240031 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231723 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 18 06:22:20.240031 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231727 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 18 06:22:20.240031 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231731 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 18 06:22:20.240031 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231736 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 18 06:22:20.240031 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231739 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 18 06:22:20.240031 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231743 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 18 06:22:20.240031 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:20.231747 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 18 06:22:20.240031 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.231755 2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 18 06:22:20.240031 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.231895 2567 server.go:962] "Client rotation is on, will bootstrap in background" Sep 18 06:22:20.240031 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.234780 2567 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 18 06:22:20.240031 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.235770 2567 server.go:1019] "Starting client certificate rotation" Sep 18 06:22:20.240031 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.235870 2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 18 06:22:20.240031 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.235900 2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 18 06:22:20.240607 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.238606 2567 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 18 06:22:20.240996 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.240976 2567 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 18 06:22:20.253952 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.253934 2567 log.go:25] "Validated CRI v1 runtime API" Sep 18 06:22:20.257792 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.257777 2567 log.go:25] "Validated CRI v1 image API" Sep 18 06:22:20.258733 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.258721 2567 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 18 06:22:20.261050 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.261028 2567 fs.go:135] Filesystem UUIDs: map[080fd2a3-4b6b-4163-8294-7c3aab73156c:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 c3c6ee40-5b92-407d-b37b-7609556fc887:/dev/nvme0n1p4] Sep 18 06:22:20.261125 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.261049 2567 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 18 06:22:20.266904 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.266803 2567 manager.go:216] Machine: {Timestamp:2026-09-18 06:22:20.266253855 +0000 UTC m=+2.005670042 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3092111 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23f99ed9e66a6619cc23e96cd28d7d SystemUUID:ec23f99e-d9e6-6a66-19cc-23e96cd28d7d BootID:4982ea3a-c620-4778-863b-58d971fc12d6 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:cb:1b:27:e7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:cb:1b:27:e7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8e:8a:63:d9:b1:fd Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 18 06:22:20.266904 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.266898 2567 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 18 06:22:20.267021 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.266966 2567 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 18 06:22:20.268816 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.268780 2567 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 18 06:22:20.268965 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.268825 2567 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-104.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 18 06:22:20.270135 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.270100 2567 topology_manager.go:138] "Creating topology manager with none policy" Sep 18 06:22:20.270234 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.270178 2567 container_manager_linux.go:309] "Creating device plugin manager" Sep 18 06:22:20.270234 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.270197 2567 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 18 06:22:20.270345 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.270244 2567 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 18 06:22:20.270415 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.270386 2567 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 18 06:22:20.274902 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.274886 2567 state_mem.go:36] "Initialized new in-memory state store" Sep 18 06:22:20.275010 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.274987 2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 18 06:22:20.275061 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.275051 2567 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 18 06:22:20.275093 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.275083 2567 kubelet.go:486] "Attempting to sync node with API server" Sep 18 06:22:20.275121 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.275097 2567 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 18 06:22:20.275121 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.275111 2567 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 18 06:22:20.275198 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.275123 2567 kubelet.go:398] "Adding apiserver pod source" Sep 18 06:22:20.275198 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.275135 2567 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 18 06:22:20.275480 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.275469 2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 18 06:22:20.275538 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.275491 2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 18 06:22:20.277484 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.277469 2567 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1" Sep 18 06:22:20.281717 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.279548 2567 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 18 06:22:20.281717 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.279576 2567 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 18 06:22:20.281717 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.280489 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 18 06:22:20.281717 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.280503 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 18 06:22:20.281717 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.280509 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 18 06:22:20.281717 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.280514 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 18 06:22:20.281717 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.280522 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 18 06:22:20.281717 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.280527 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 18 06:22:20.281717 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.280540 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 18 06:22:20.281717 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.280545 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 18 06:22:20.281717 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.280552 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 18 06:22:20.281717 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.280557 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 18 06:22:20.281717 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.280565 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 18 06:22:20.281717 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.280570 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 18 06:22:20.281717 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.280590 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 18 06:22:20.281717 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.280595 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 18 06:22:20.284357 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.284346 2567 server.go:1268] "Started kubelet" Sep 18 06:22:20.284430 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.284407 2567 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 18 06:22:20.284486 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.284440 2567 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 18 06:22:20.284557 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.284545 2567 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 18 06:22:20.285127 ip-10-0-142-104 systemd[1]: Started Kubernetes Kubelet. Sep 18 06:22:20.286209 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.286087 2567 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 18 06:22:20.286727 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.286710 2567 server.go:310] "Adding debug handlers to kubelet server" Sep 18 06:22:20.288559 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.288542 2567 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-104.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 06:22:20.289016 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:20.288990 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 18 06:22:20.289016 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:20.288993 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-104.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 18 06:22:20.289467 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:20.289451 2567 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 18 06:22:20.292198 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.292183 2567 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 18 06:22:20.292277 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.292265 2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 18 06:22:20.292611 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:20.292399 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-104.ec2.internal\" not found" Sep 18 06:22:20.292695 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.292688 2567 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 18 06:22:20.292761 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.292706 2567 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 18 06:22:20.292821 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.292769 2567 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 18 06:22:20.292891 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.292876 2567 reconstruct.go:97] "Volume reconstruction finished" Sep 18 06:22:20.292891 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.292891 2567 reconciler.go:29] "Reconciler: start to sync state" Sep 18 06:22:20.294376 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.294330 2567 factory.go:55] Registering systemd factory Sep 18 06:22:20.294376 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.294346 2567 factory.go:223] Registration of the systemd container factory successfully Sep 18 06:22:20.294574 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.294562 2567 factory.go:153] Registering CRI-O factory Sep 18 06:22:20.294641 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.294580 2567 factory.go:223] Registration of the crio container factory successfully Sep 18 06:22:20.294641 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.294635 2567 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 18 06:22:20.294741 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.294653 2567 factory.go:103] Registering Raw factory Sep 18 06:22:20.294741 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.294670 2567 manager.go:1196] Started watching for new ooms in manager Sep 18 06:22:20.294741 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:20.294706 2567 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-104.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 18 06:22:20.294741 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:20.294731 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 18 06:22:20.295037 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.295024 2567 manager.go:349] Starting recovery of all containers Sep 18 06:22:20.296452 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.296431 2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dccm4" Sep 18 06:22:20.318379 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.318338 2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dccm4" Sep 18 06:22:20.324656 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.324642 2567 manager.go:354] Recovery completed Sep 18 06:22:20.328627 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.328614 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 06:22:20.330787 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.330770 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-104.ec2.internal" event="NodeHasSufficientMemory" Sep 18 06:22:20.330877 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.330797 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-104.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 06:22:20.330877 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.330808 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-104.ec2.internal" event="NodeHasSufficientPID" Sep 18 06:22:20.331379 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.331347 2567 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 18 06:22:20.331379 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.331359 2567 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 18 06:22:20.331491 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.331391 2567 state_mem.go:36] "Initialized new in-memory state store" Sep 18 06:22:20.331491 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.331392 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 06:22:20.333421 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.333408 2567 policy_none.go:49] "None policy: Start" Sep 18 06:22:20.333421 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.333415 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-104.ec2.internal" event="NodeHasSufficientMemory" Sep 18 06:22:20.333515 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.333427 2567 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 18 06:22:20.333515 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.333448 2567 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 18 06:22:20.333515 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.333449 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-104.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 06:22:20.333515 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.333473 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-104.ec2.internal" event="NodeHasSufficientPID" Sep 18 06:22:20.335497 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.335484 2567 policy_none.go:47] "Start" Sep 18 06:22:20.370039 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.369894 2567 manager.go:337] "Starting Device Plugin manager" Sep 18 06:22:20.389228 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:20.370060 2567 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 18 06:22:20.389228 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.370074 2567 server.go:87] "Starting device plugin registration server" Sep 18 06:22:20.389228 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.370321 2567 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 18 06:22:20.389228 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.370347 2567 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 18 06:22:20.389228 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.370473 2567 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 18 06:22:20.389228 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.370557 2567 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 18 06:22:20.389228 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.370563 2567 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 18 06:22:20.389228 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:20.370959 2567 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 18 06:22:20.389228 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:20.371010 2567 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-104.ec2.internal\" not found" Sep 18 06:22:20.437789 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.437761 2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 18 06:22:20.439016 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.438994 2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 18 06:22:20.439084 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.439059 2567 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 18 06:22:20.439084 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.439082 2567 kubelet.go:2448] "Starting kubelet main sync loop" Sep 18 06:22:20.439165 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:20.439117 2567 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 18 06:22:20.441333 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.441317 2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 18 06:22:20.470533 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.470476 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 06:22:20.471262 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.471245 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-104.ec2.internal" event="NodeHasSufficientMemory" Sep 18 06:22:20.471318 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.471273 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-104.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 06:22:20.471318 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.471286 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-104.ec2.internal" event="NodeHasSufficientPID" Sep 18 06:22:20.471318 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.471306 2567 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-104.ec2.internal" Sep 18 06:22:20.480502 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.480488 2567 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-104.ec2.internal" Sep 18 06:22:20.480545 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:20.480508 2567 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-104.ec2.internal\": node \"ip-10-0-142-104.ec2.internal\" not found" Sep 18 06:22:20.495794 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:20.495775 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-104.ec2.internal\" not found" Sep 18 06:22:20.540229 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.540203 2567 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-142-104.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-104.ec2.internal"] Sep 18 06:22:20.540289 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.540260 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 06:22:20.540969 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.540953 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-104.ec2.internal" event="NodeHasSufficientMemory" Sep 18 06:22:20.541037 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.540982 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-104.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 06:22:20.541037 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.540993 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-104.ec2.internal" event="NodeHasSufficientPID" Sep 18 06:22:20.543197 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.543186 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 06:22:20.543323 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.543303 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-104.ec2.internal" Sep 18 06:22:20.543423 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.543332 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 06:22:20.543800 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.543787 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-104.ec2.internal" event="NodeHasSufficientMemory" Sep 18 06:22:20.543846 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.543815 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-104.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 06:22:20.543846 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.543830 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-104.ec2.internal" event="NodeHasSufficientPID" Sep 18 06:22:20.544224 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.544209 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-104.ec2.internal" event="NodeHasSufficientMemory" Sep 18 06:22:20.544300 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.544239 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-104.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 06:22:20.544300 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.544254 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-104.ec2.internal" event="NodeHasSufficientPID" Sep 18 06:22:20.545943 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.545929 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-104.ec2.internal" Sep 18 06:22:20.545997 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.545960 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 06:22:20.546652 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.546636 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-104.ec2.internal" event="NodeHasSufficientMemory" Sep 18 06:22:20.546714 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.546662 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-104.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 06:22:20.546714 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.546671 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-104.ec2.internal" event="NodeHasSufficientPID" Sep 18 06:22:20.568871 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:20.568853 2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-104.ec2.internal\" not found" node="ip-10-0-142-104.ec2.internal" Sep 18 06:22:20.572918 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:20.572904 2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-104.ec2.internal\" not found" node="ip-10-0-142-104.ec2.internal" Sep 18 06:22:20.593849 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.593831 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/093f6c5a5a5dfa9d45afb8587676bdd0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-104.ec2.internal\" (UID: \"093f6c5a5a5dfa9d45afb8587676bdd0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-104.ec2.internal" Sep 18 06:22:20.593898 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.593853 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/093f6c5a5a5dfa9d45afb8587676bdd0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-104.ec2.internal\" (UID: \"093f6c5a5a5dfa9d45afb8587676bdd0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-104.ec2.internal" Sep 18 06:22:20.593898 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.593868 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f23d030455f844c3f1c0ef497274c121-config\") pod \"kube-apiserver-proxy-ip-10-0-142-104.ec2.internal\" (UID: \"f23d030455f844c3f1c0ef497274c121\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-104.ec2.internal" Sep 18 06:22:20.595859 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:20.595847 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-104.ec2.internal\" not found" Sep 18 06:22:20.694854 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.694833 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/093f6c5a5a5dfa9d45afb8587676bdd0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-104.ec2.internal\" (UID: \"093f6c5a5a5dfa9d45afb8587676bdd0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-104.ec2.internal" Sep 18 06:22:20.694925 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.694855 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f23d030455f844c3f1c0ef497274c121-config\") pod \"kube-apiserver-proxy-ip-10-0-142-104.ec2.internal\" (UID: \"f23d030455f844c3f1c0ef497274c121\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-104.ec2.internal" Sep 18 06:22:20.694925 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.694871 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/093f6c5a5a5dfa9d45afb8587676bdd0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-104.ec2.internal\" (UID: \"093f6c5a5a5dfa9d45afb8587676bdd0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-104.ec2.internal" Sep 18 06:22:20.694925 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.694895 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/093f6c5a5a5dfa9d45afb8587676bdd0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-104.ec2.internal\" (UID: \"093f6c5a5a5dfa9d45afb8587676bdd0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-104.ec2.internal" Sep 18 06:22:20.695038 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.694928 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/093f6c5a5a5dfa9d45afb8587676bdd0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-104.ec2.internal\" (UID: \"093f6c5a5a5dfa9d45afb8587676bdd0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-104.ec2.internal" Sep 18 06:22:20.695038 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.694934 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f23d030455f844c3f1c0ef497274c121-config\") pod \"kube-apiserver-proxy-ip-10-0-142-104.ec2.internal\" (UID: \"f23d030455f844c3f1c0ef497274c121\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-104.ec2.internal" Sep 18 06:22:20.696942 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:20.696928 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-104.ec2.internal\" not found" Sep 18 06:22:20.797691 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:20.797631 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-104.ec2.internal\" not found" Sep 18 06:22:20.872864 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.872838 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-104.ec2.internal" Sep 18 06:22:20.876402 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:20.876386 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-104.ec2.internal" Sep 18 06:22:20.898631 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:20.898608 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-104.ec2.internal\" not found" Sep 18 06:22:20.999142 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:20.999094 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-104.ec2.internal\" not found" Sep 18 06:22:21.099593 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:21.099539 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-104.ec2.internal\" not found" Sep 18 06:22:21.117703 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:21.117682 2567 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 18 06:22:21.200057 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:21.200031 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-104.ec2.internal\" not found" Sep 18 06:22:21.235205 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:21.235180 2567 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 18 06:22:21.235630 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:21.235299 2567 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 18 06:22:21.235630 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:21.235305 2567 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 18 06:22:21.259968 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:21.259948 2567 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 18 06:22:21.300448 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:21.300425 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-104.ec2.internal\" not found" Sep 18 06:22:21.324602 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:21.324564 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-17 06:17:20 +0000 UTC" deadline="2028-05-26 18:43:33.641832212 +0000 UTC" Sep 18 06:22:21.324602 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:21.324601 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14796h21m12.317234451s" Sep 18 06:22:21.401081 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:21.401034 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-104.ec2.internal\" not found" Sep 18 06:22:21.464382 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:21.464345 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf23d030455f844c3f1c0ef497274c121.slice/crio-64127663be7c8084c31c8f475955aeae39346f3306e45f44bdf59848162755bf WatchSource:0}: Error finding container 64127663be7c8084c31c8f475955aeae39346f3306e45f44bdf59848162755bf: Status 404 returned error can't find the container with id 64127663be7c8084c31c8f475955aeae39346f3306e45f44bdf59848162755bf Sep 18 06:22:21.464696 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:21.464670 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod093f6c5a5a5dfa9d45afb8587676bdd0.slice/crio-e36caf5ed2911b18b8413f60581995f10fbaab2a0eec8605de7ac361c609e3b6 WatchSource:0}: Error finding container e36caf5ed2911b18b8413f60581995f10fbaab2a0eec8605de7ac361c609e3b6: Status 404 returned error can't find the container with id e36caf5ed2911b18b8413f60581995f10fbaab2a0eec8605de7ac361c609e3b6 Sep 18 06:22:21.469355 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:21.469339 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 18 06:22:21.502117 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:21.502094 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-104.ec2.internal\" not found" Sep 18 06:22:21.602557 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:21.602536 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-104.ec2.internal\" not found" Sep 18 06:22:21.703030 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:21.702985 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-104.ec2.internal\" not found" Sep 18 06:22:21.803583 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:21.803553 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-104.ec2.internal\" not found" Sep 18 06:22:21.804628 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:21.804600 2567 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 18 06:22:21.893547 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:21.893516 2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-104.ec2.internal" Sep 18 06:22:21.902401 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:21.902358 2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 18 06:22:21.903460 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:21.903439 2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-104.ec2.internal" Sep 18 06:22:21.912286 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:21.911816 2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 18 06:22:22.276740 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.276713 2567 apiserver.go:52] "Watching apiserver" Sep 18 06:22:22.285785 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.285756 2567 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 18 06:22:22.286171 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.286148 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmmq2","openshift-dns/node-resolver-kgx9w","openshift-image-registry/node-ca-xccgg","openshift-network-operator/iptables-alerter-p94t6","openshift-ovn-kubernetes/ovnkube-node-hjr4h","kube-system/konnectivity-agent-sxrxx","kube-system/kube-apiserver-proxy-ip-10-0-142-104.ec2.internal","openshift-cluster-node-tuning-operator/tuned-w7cmt","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-104.ec2.internal","openshift-multus/multus-5dt8f","openshift-multus/multus-additional-cni-plugins-q4gmz","openshift-multus/network-metrics-daemon-k77fk","openshift-network-diagnostics/network-check-target-5rhpx","kube-system/global-pull-secret-syncer-2jzcv"] Sep 18 06:22:22.288823 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.288797 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5dt8f" Sep 18 06:22:22.290974 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.290951 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5rhpx" Sep 18 06:22:22.291087 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:22.291030 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5rhpx" podUID="ce5f0b70-0633-4019-a805-85d798616ac2" Sep 18 06:22:22.291281 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.291262 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-nrw2l\"" Sep 18 06:22:22.291406 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.291381 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 18 06:22:22.291495 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.291477 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 18 06:22:22.291571 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.291498 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 18 06:22:22.291663 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.291643 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 18 06:22:22.293146 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.293124 2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 18 06:22:22.293891 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.293476 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xccgg" Sep 18 06:22:22.295718 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.295455 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-p94t6" Sep 18 06:22:22.295718 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.295522 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-w5zhk\"" Sep 18 06:22:22.295912 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.295895 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 18 06:22:22.296067 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.296042 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 18 06:22:22.296067 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.296052 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 18 06:22:22.297662 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.297646 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 18 06:22:22.297960 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.297941 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2mkml\"" Sep 18 06:22:22.298202 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.298187 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 18 06:22:22.298456 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.298190 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:22.298776 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.298757 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 18 06:22:22.300056 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.300039 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sxrxx" Sep 18 06:22:22.300897 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.300881 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 18 06:22:22.301316 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.301135 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 18 06:22:22.301316 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.301139 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-gc69j\"" Sep 18 06:22:22.301316 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.301262 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 18 06:22:22.301538 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.301328 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 18 06:22:22.302470 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.302449 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 18 06:22:22.302567 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.302509 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 18 06:22:22.302839 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.302822 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 18 06:22:22.302900 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.302853 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2jzcv" Sep 18 06:22:22.302900 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.302870 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-cqdmx\"" Sep 18 06:22:22.302994 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:22.302918 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2jzcv" podUID="e57e7a14-2ca6-4f41-875f-822739fa1c43" Sep 18 06:22:22.303262 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.303112 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 18 06:22:22.304268 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.304246 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/54e09508-bf0d-4f03-b35e-69b4b61d3a9f-multus-cni-dir\") pod \"multus-5dt8f\" (UID: \"54e09508-bf0d-4f03-b35e-69b4b61d3a9f\") " pod="openshift-multus/multus-5dt8f" Sep 18 06:22:22.304383 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.304274 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/54e09508-bf0d-4f03-b35e-69b4b61d3a9f-cnibin\") pod \"multus-5dt8f\" (UID: \"54e09508-bf0d-4f03-b35e-69b4b61d3a9f\") " pod="openshift-multus/multus-5dt8f" Sep 18 06:22:22.304383 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.304296 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/54e09508-bf0d-4f03-b35e-69b4b61d3a9f-host-run-k8s-cni-cncf-io\") pod \"multus-5dt8f\" (UID: \"54e09508-bf0d-4f03-b35e-69b4b61d3a9f\") " pod="openshift-multus/multus-5dt8f" Sep 18 06:22:22.304383 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.304320 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/54e09508-bf0d-4f03-b35e-69b4b61d3a9f-host-var-lib-cni-bin\") pod \"multus-5dt8f\" (UID: \"54e09508-bf0d-4f03-b35e-69b4b61d3a9f\") " pod="openshift-multus/multus-5dt8f" Sep 18 06:22:22.304383 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.304356 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/54e09508-bf0d-4f03-b35e-69b4b61d3a9f-host-var-lib-kubelet\") pod \"multus-5dt8f\" (UID: \"54e09508-bf0d-4f03-b35e-69b4b61d3a9f\") " pod="openshift-multus/multus-5dt8f" Sep 18 06:22:22.304586 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.304397 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/54e09508-bf0d-4f03-b35e-69b4b61d3a9f-system-cni-dir\") pod \"multus-5dt8f\" (UID: \"54e09508-bf0d-4f03-b35e-69b4b61d3a9f\") " pod="openshift-multus/multus-5dt8f" Sep 18 06:22:22.304586 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.304423 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/54e09508-bf0d-4f03-b35e-69b4b61d3a9f-multus-socket-dir-parent\") pod \"multus-5dt8f\" (UID: \"54e09508-bf0d-4f03-b35e-69b4b61d3a9f\") " pod="openshift-multus/multus-5dt8f" Sep 18 06:22:22.304586 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.304446 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/54e09508-bf0d-4f03-b35e-69b4b61d3a9f-hostroot\") pod \"multus-5dt8f\" (UID: \"54e09508-bf0d-4f03-b35e-69b4b61d3a9f\") " pod="openshift-multus/multus-5dt8f" Sep 18 06:22:22.304586 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.304467 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/54e09508-bf0d-4f03-b35e-69b4b61d3a9f-multus-conf-dir\") pod \"multus-5dt8f\" (UID: \"54e09508-bf0d-4f03-b35e-69b4b61d3a9f\") " pod="openshift-multus/multus-5dt8f" Sep 18 06:22:22.304586 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.304576 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/54e09508-bf0d-4f03-b35e-69b4b61d3a9f-multus-daemon-config\") pod \"multus-5dt8f\" (UID: \"54e09508-bf0d-4f03-b35e-69b4b61d3a9f\") " pod="openshift-multus/multus-5dt8f" Sep 18 06:22:22.304821 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.304609 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/54e09508-bf0d-4f03-b35e-69b4b61d3a9f-host-run-multus-certs\") pod \"multus-5dt8f\" (UID: \"54e09508-bf0d-4f03-b35e-69b4b61d3a9f\") " pod="openshift-multus/multus-5dt8f" Sep 18 06:22:22.304821 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.304581 2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 18 06:22:22.304821 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.304658 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/54e09508-bf0d-4f03-b35e-69b4b61d3a9f-etc-kubernetes\") pod \"multus-5dt8f\" (UID: \"54e09508-bf0d-4f03-b35e-69b4b61d3a9f\") " pod="openshift-multus/multus-5dt8f" Sep 18 06:22:22.304821 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.304715 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/27cccfd2-ff8e-4f71-bc24-5072f09b7f21-host\") pod \"node-ca-xccgg\" (UID: \"27cccfd2-ff8e-4f71-bc24-5072f09b7f21\") " pod="openshift-image-registry/node-ca-xccgg" Sep 18 06:22:22.304821 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.304739 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/54e09508-bf0d-4f03-b35e-69b4b61d3a9f-host-run-netns\") pod \"multus-5dt8f\" (UID: \"54e09508-bf0d-4f03-b35e-69b4b61d3a9f\") " pod="openshift-multus/multus-5dt8f" Sep 18 06:22:22.304821 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.304794 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/54e09508-bf0d-4f03-b35e-69b4b61d3a9f-host-var-lib-cni-multus\") pod \"multus-5dt8f\" (UID: \"54e09508-bf0d-4f03-b35e-69b4b61d3a9f\") " pod="openshift-multus/multus-5dt8f" Sep 18 06:22:22.305092 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.304868 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwmtn\" (UniqueName: \"kubernetes.io/projected/ce5f0b70-0633-4019-a805-85d798616ac2-kube-api-access-pwmtn\") pod \"network-check-target-5rhpx\" (UID: \"ce5f0b70-0633-4019-a805-85d798616ac2\") " pod="openshift-network-diagnostics/network-check-target-5rhpx" Sep 18 06:22:22.305092 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.304898 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-w7cmt" Sep 18 06:22:22.305092 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.304950 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/27cccfd2-ff8e-4f71-bc24-5072f09b7f21-serviceca\") pod \"node-ca-xccgg\" (UID: \"27cccfd2-ff8e-4f71-bc24-5072f09b7f21\") " pod="openshift-image-registry/node-ca-xccgg" Sep 18 06:22:22.305092 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.305017 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9c2fc9d9-19db-4703-aa25-95b5f5a42e99-iptables-alerter-script\") pod \"iptables-alerter-p94t6\" (UID: \"9c2fc9d9-19db-4703-aa25-95b5f5a42e99\") " pod="openshift-network-operator/iptables-alerter-p94t6" Sep 18 06:22:22.305092 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.305044 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/54e09508-bf0d-4f03-b35e-69b4b61d3a9f-cni-binary-copy\") pod \"multus-5dt8f\" (UID: \"54e09508-bf0d-4f03-b35e-69b4b61d3a9f\") " pod="openshift-multus/multus-5dt8f" Sep 18 06:22:22.305298 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.305089 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slppz\" (UniqueName: \"kubernetes.io/projected/54e09508-bf0d-4f03-b35e-69b4b61d3a9f-kube-api-access-slppz\") pod \"multus-5dt8f\" (UID: \"54e09508-bf0d-4f03-b35e-69b4b61d3a9f\") " pod="openshift-multus/multus-5dt8f" Sep 18 06:22:22.305298 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.305116 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvsvc\" (UniqueName: \"kubernetes.io/projected/27cccfd2-ff8e-4f71-bc24-5072f09b7f21-kube-api-access-lvsvc\") pod \"node-ca-xccgg\" (UID: \"27cccfd2-ff8e-4f71-bc24-5072f09b7f21\") " pod="openshift-image-registry/node-ca-xccgg" Sep 18 06:22:22.305298 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.305138 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9c2fc9d9-19db-4703-aa25-95b5f5a42e99-host-slash\") pod \"iptables-alerter-p94t6\" (UID: \"9c2fc9d9-19db-4703-aa25-95b5f5a42e99\") " pod="openshift-network-operator/iptables-alerter-p94t6" Sep 18 06:22:22.305298 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.305161 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7b74\" (UniqueName: \"kubernetes.io/projected/9c2fc9d9-19db-4703-aa25-95b5f5a42e99-kube-api-access-l7b74\") pod \"iptables-alerter-p94t6\" (UID: \"9c2fc9d9-19db-4703-aa25-95b5f5a42e99\") " pod="openshift-network-operator/iptables-alerter-p94t6" Sep 18 06:22:22.307187 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.306883 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 18 06:22:22.307187 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.306972 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jm5hn\"" Sep 18 06:22:22.307321 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.307247 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 18 06:22:22.308125 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.308105 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kgx9w" Sep 18 06:22:22.310505 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.310398 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 18 06:22:22.310660 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.310641 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 18 06:22:22.310847 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.310805 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-t779k\"" Sep 18 06:22:22.312139 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.311867 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmmq2" Sep 18 06:22:22.313811 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.313779 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 18 06:22:22.313911 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.313865 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 18 06:22:22.314408 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.314391 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 18 06:22:22.314529 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.314511 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xbl5k\"" Sep 18 06:22:22.314995 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.314975 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-q4gmz" Sep 18 06:22:22.317309 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.317291 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 18 06:22:22.317439 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.317311 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k77fk" Sep 18 06:22:22.317439 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:22.317360 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k77fk" podUID="66dca255-1289-49ec-b200-e950a30e3d9c" Sep 18 06:22:22.317439 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.317423 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xbvln\"" Sep 18 06:22:22.317585 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.317512 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 18 06:22:22.325000 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.324981 2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-8bqm9" Sep 18 06:22:22.332662 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.332643 2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-8bqm9" Sep 18 06:22:22.393979 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.393952 2567 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 18 06:22:22.405418 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.405394 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62ffb\" (UniqueName: \"kubernetes.io/projected/4d3101c6-973c-48d6-8efe-f4fab4282824-kube-api-access-62ffb\") pod \"aws-ebs-csi-driver-node-wmmq2\" (UID: \"4d3101c6-973c-48d6-8efe-f4fab4282824\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmmq2" Sep 18 06:22:22.405531 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.405437 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb059951-e263-40da-9155-4fd0cf242820-host-kubelet\") pod \"ovnkube-node-hjr4h\" (UID: \"eb059951-e263-40da-9155-4fd0cf242820\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:22.405531 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.405463 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb059951-e263-40da-9155-4fd0cf242820-etc-openvswitch\") pod \"ovnkube-node-hjr4h\" (UID: \"eb059951-e263-40da-9155-4fd0cf242820\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:22.405531 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.405494 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzdct\" (UniqueName: \"kubernetes.io/projected/eb059951-e263-40da-9155-4fd0cf242820-kube-api-access-gzdct\") pod \"ovnkube-node-hjr4h\" (UID: \"eb059951-e263-40da-9155-4fd0cf242820\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:22.405657 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.405537 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/54e09508-bf0d-4f03-b35e-69b4b61d3a9f-system-cni-dir\") pod \"multus-5dt8f\" (UID: \"54e09508-bf0d-4f03-b35e-69b4b61d3a9f\") " pod="openshift-multus/multus-5dt8f" Sep 18 06:22:22.405657 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.405570 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/54e09508-bf0d-4f03-b35e-69b4b61d3a9f-hostroot\") pod \"multus-5dt8f\" (UID: \"54e09508-bf0d-4f03-b35e-69b4b61d3a9f\") " pod="openshift-multus/multus-5dt8f" Sep 18 06:22:22.405657 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.405600 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/54e09508-bf0d-4f03-b35e-69b4b61d3a9f-system-cni-dir\") pod \"multus-5dt8f\" (UID: \"54e09508-bf0d-4f03-b35e-69b4b61d3a9f\") " pod="openshift-multus/multus-5dt8f" Sep 18 06:22:22.405657 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.405637 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12-run\") pod \"tuned-w7cmt\" (UID: \"e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12\") " pod="openshift-cluster-node-tuning-operator/tuned-w7cmt" Sep 18 06:22:22.405833 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.405673 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4d3101c6-973c-48d6-8efe-f4fab4282824-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wmmq2\" (UID: \"4d3101c6-973c-48d6-8efe-f4fab4282824\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmmq2" Sep 18 06:22:22.405833 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.405711 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4d3101c6-973c-48d6-8efe-f4fab4282824-socket-dir\") pod \"aws-ebs-csi-driver-node-wmmq2\" (UID: \"4d3101c6-973c-48d6-8efe-f4fab4282824\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmmq2" Sep 18 06:22:22.405833 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.405704 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/54e09508-bf0d-4f03-b35e-69b4b61d3a9f-hostroot\") pod \"multus-5dt8f\" (UID: \"54e09508-bf0d-4f03-b35e-69b4b61d3a9f\") " pod="openshift-multus/multus-5dt8f" Sep 18 06:22:22.405833 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.405748 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eb059951-e263-40da-9155-4fd0cf242820-run-ovn\") pod \"ovnkube-node-hjr4h\" (UID: \"eb059951-e263-40da-9155-4fd0cf242820\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:22.405833 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.405774 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/54e09508-bf0d-4f03-b35e-69b4b61d3a9f-host-run-netns\") pod \"multus-5dt8f\" (UID: \"54e09508-bf0d-4f03-b35e-69b4b61d3a9f\") " pod="openshift-multus/multus-5dt8f" Sep 18 06:22:22.405833 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.405805 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwmtn\" (UniqueName: \"kubernetes.io/projected/ce5f0b70-0633-4019-a805-85d798616ac2-kube-api-access-pwmtn\") pod \"network-check-target-5rhpx\" (UID: \"ce5f0b70-0633-4019-a805-85d798616ac2\") " pod="openshift-network-diagnostics/network-check-target-5rhpx" Sep 18 06:22:22.405833 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.405815 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/54e09508-bf0d-4f03-b35e-69b4b61d3a9f-host-run-netns\") pod \"multus-5dt8f\" (UID: \"54e09508-bf0d-4f03-b35e-69b4b61d3a9f\") " pod="openshift-multus/multus-5dt8f" Sep 18 06:22:22.406078 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.405840 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0b83c068-4913-4ee5-85ed-17f44dc9f0ca-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q4gmz\" (UID: \"0b83c068-4913-4ee5-85ed-17f44dc9f0ca\") " pod="openshift-multus/multus-additional-cni-plugins-q4gmz" Sep 18 06:22:22.406078 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.405866 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7e27a94-0a48-466c-abdb-9f6b0758e358-tmp-dir\") pod \"node-resolver-kgx9w\" (UID: \"c7e27a94-0a48-466c-abdb-9f6b0758e358\") " pod="openshift-dns/node-resolver-kgx9w" Sep 18 06:22:22.406078 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.405889 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66dca255-1289-49ec-b200-e950a30e3d9c-metrics-certs\") pod \"network-metrics-daemon-k77fk\" (UID: \"66dca255-1289-49ec-b200-e950a30e3d9c\") " pod="openshift-multus/network-metrics-daemon-k77fk" Sep 18 06:22:22.406078 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.405915 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb059951-e263-40da-9155-4fd0cf242820-host-cni-bin\") pod \"ovnkube-node-hjr4h\" (UID: \"eb059951-e263-40da-9155-4fd0cf242820\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:22.406078 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.405954 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/631676f2-32af-4b0d-834b-41c7c59db5ba-konnectivity-ca\") pod \"konnectivity-agent-sxrxx\" (UID: \"631676f2-32af-4b0d-834b-41c7c59db5ba\") " pod="kube-system/konnectivity-agent-sxrxx" Sep 18 06:22:22.406078 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.405995 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb059951-e263-40da-9155-4fd0cf242820-host-slash\") pod \"ovnkube-node-hjr4h\" (UID: \"eb059951-e263-40da-9155-4fd0cf242820\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:22.406078 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.406019 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eb059951-e263-40da-9155-4fd0cf242820-run-systemd\") pod \"ovnkube-node-hjr4h\" (UID: \"eb059951-e263-40da-9155-4fd0cf242820\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:22.406078 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.406048 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9c2fc9d9-19db-4703-aa25-95b5f5a42e99-host-slash\") pod \"iptables-alerter-p94t6\" (UID: \"9c2fc9d9-19db-4703-aa25-95b5f5a42e99\") " pod="openshift-network-operator/iptables-alerter-p94t6" Sep 18 06:22:22.406392 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.406112 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9c2fc9d9-19db-4703-aa25-95b5f5a42e99-host-slash\") pod \"iptables-alerter-p94t6\" (UID: \"9c2fc9d9-19db-4703-aa25-95b5f5a42e99\") " pod="openshift-network-operator/iptables-alerter-p94t6" Sep 18 06:22:22.406392 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.406127 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7b74\" (UniqueName: \"kubernetes.io/projected/9c2fc9d9-19db-4703-aa25-95b5f5a42e99-kube-api-access-l7b74\") pod \"iptables-alerter-p94t6\" (UID: \"9c2fc9d9-19db-4703-aa25-95b5f5a42e99\") " pod="openshift-network-operator/iptables-alerter-p94t6" Sep 18 06:22:22.406392 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.406165 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/54e09508-bf0d-4f03-b35e-69b4b61d3a9f-multus-cni-dir\") pod \"multus-5dt8f\" (UID: \"54e09508-bf0d-4f03-b35e-69b4b61d3a9f\") " pod="openshift-multus/multus-5dt8f" Sep 18 06:22:22.406392 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.406191 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/54e09508-bf0d-4f03-b35e-69b4b61d3a9f-cnibin\") pod \"multus-5dt8f\" (UID: \"54e09508-bf0d-4f03-b35e-69b4b61d3a9f\") " pod="openshift-multus/multus-5dt8f" Sep 18 06:22:22.406392 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.406216 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/54e09508-bf0d-4f03-b35e-69b4b61d3a9f-host-run-k8s-cni-cncf-io\") pod \"multus-5dt8f\" (UID: \"54e09508-bf0d-4f03-b35e-69b4b61d3a9f\") " pod="openshift-multus/multus-5dt8f" Sep 18 06:22:22.406392 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.406240 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/54e09508-bf0d-4f03-b35e-69b4b61d3a9f-multus-daemon-config\") pod \"multus-5dt8f\" (UID: \"54e09508-bf0d-4f03-b35e-69b4b61d3a9f\") " pod="openshift-multus/multus-5dt8f" Sep 18 06:22:22.406392 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.406267 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/54e09508-bf0d-4f03-b35e-69b4b61d3a9f-cnibin\") pod \"multus-5dt8f\" (UID: \"54e09508-bf0d-4f03-b35e-69b4b61d3a9f\") " pod="openshift-multus/multus-5dt8f" Sep 18 06:22:22.406392 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.406265 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e57e7a14-2ca6-4f41-875f-822739fa1c43-dbus\") pod \"global-pull-secret-syncer-2jzcv\" (UID: \"e57e7a14-2ca6-4f41-875f-822739fa1c43\") " pod="kube-system/global-pull-secret-syncer-2jzcv" Sep 18 06:22:22.406392 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.406310 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/54e09508-bf0d-4f03-b35e-69b4b61d3a9f-host-run-k8s-cni-cncf-io\") pod \"multus-5dt8f\" (UID: \"54e09508-bf0d-4f03-b35e-69b4b61d3a9f\") " pod="openshift-multus/multus-5dt8f" Sep 18 06:22:22.406392 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.406316 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eb059951-e263-40da-9155-4fd0cf242820-ovn-node-metrics-cert\") pod \"ovnkube-node-hjr4h\" (UID: \"eb059951-e263-40da-9155-4fd0cf242820\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:22.406392 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.406339 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/54e09508-bf0d-4f03-b35e-69b4b61d3a9f-multus-conf-dir\") pod \"multus-5dt8f\" (UID: \"54e09508-bf0d-4f03-b35e-69b4b61d3a9f\") " pod="openshift-multus/multus-5dt8f" Sep 18 06:22:22.406392 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.406354 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4d3101c6-973c-48d6-8efe-f4fab4282824-sys-fs\") pod \"aws-ebs-csi-driver-node-wmmq2\" (UID: \"4d3101c6-973c-48d6-8efe-f4fab4282824\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmmq2" Sep 18 06:22:22.406392 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.406389 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sct85\" (UniqueName: \"kubernetes.io/projected/66dca255-1289-49ec-b200-e950a30e3d9c-kube-api-access-sct85\") pod \"network-metrics-daemon-k77fk\" (UID: \"66dca255-1289-49ec-b200-e950a30e3d9c\") " pod="openshift-multus/network-metrics-daemon-k77fk" Sep 18 06:22:22.407017 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.406412 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/54e09508-bf0d-4f03-b35e-69b4b61d3a9f-host-var-lib-cni-multus\") pod \"multus-5dt8f\" (UID: \"54e09508-bf0d-4f03-b35e-69b4b61d3a9f\") " pod="openshift-multus/multus-5dt8f" Sep 18 06:22:22.407017 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.406398 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/54e09508-bf0d-4f03-b35e-69b4b61d3a9f-multus-cni-dir\") pod \"multus-5dt8f\" (UID: \"54e09508-bf0d-4f03-b35e-69b4b61d3a9f\") " pod="openshift-multus/multus-5dt8f" Sep 18 06:22:22.407017 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.406428 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/54e09508-bf0d-4f03-b35e-69b4b61d3a9f-multus-conf-dir\") pod \"multus-5dt8f\" (UID: \"54e09508-bf0d-4f03-b35e-69b4b61d3a9f\") " pod="openshift-multus/multus-5dt8f" Sep 18 06:22:22.407017 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.406434 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9c2fc9d9-19db-4703-aa25-95b5f5a42e99-iptables-alerter-script\") pod \"iptables-alerter-p94t6\" (UID: \"9c2fc9d9-19db-4703-aa25-95b5f5a42e99\") " pod="openshift-network-operator/iptables-alerter-p94t6" Sep 18 06:22:22.407017 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.406453 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/54e09508-bf0d-4f03-b35e-69b4b61d3a9f-host-var-lib-cni-multus\") pod \"multus-5dt8f\" (UID: \"54e09508-bf0d-4f03-b35e-69b4b61d3a9f\") " pod="openshift-multus/multus-5dt8f" Sep 18 06:22:22.407017 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.406482 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb059951-e263-40da-9155-4fd0cf242820-var-lib-openvswitch\") pod \"ovnkube-node-hjr4h\" (UID: \"eb059951-e263-40da-9155-4fd0cf242820\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:22.407017 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.406508 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12-sys\") pod \"tuned-w7cmt\" (UID: \"e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12\") " pod="openshift-cluster-node-tuning-operator/tuned-w7cmt" Sep 18 06:22:22.407017 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.406532 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12-tmp\") pod \"tuned-w7cmt\" (UID: \"e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12\") " pod="openshift-cluster-node-tuning-operator/tuned-w7cmt" Sep 18 06:22:22.407017 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.406555 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0b83c068-4913-4ee5-85ed-17f44dc9f0ca-cnibin\") pod \"multus-additional-cni-plugins-q4gmz\" (UID: \"0b83c068-4913-4ee5-85ed-17f44dc9f0ca\") " pod="openshift-multus/multus-additional-cni-plugins-q4gmz" Sep 18 06:22:22.407017 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.406581 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4d3101c6-973c-48d6-8efe-f4fab4282824-device-dir\") pod \"aws-ebs-csi-driver-node-wmmq2\" (UID: \"4d3101c6-973c-48d6-8efe-f4fab4282824\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmmq2" Sep 18 06:22:22.407017 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.406604 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eb059951-e263-40da-9155-4fd0cf242820-systemd-units\") pod \"ovnkube-node-hjr4h\" (UID: \"eb059951-e263-40da-9155-4fd0cf242820\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:22.407017 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.406630 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/54e09508-bf0d-4f03-b35e-69b4b61d3a9f-host-var-lib-cni-bin\") pod \"multus-5dt8f\" (UID: \"54e09508-bf0d-4f03-b35e-69b4b61d3a9f\") " pod="openshift-multus/multus-5dt8f" Sep 18 06:22:22.407017 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.406653 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/54e09508-bf0d-4f03-b35e-69b4b61d3a9f-host-var-lib-kubelet\") pod \"multus-5dt8f\" (UID: \"54e09508-bf0d-4f03-b35e-69b4b61d3a9f\") " pod="openshift-multus/multus-5dt8f" Sep 18 06:22:22.407017 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.406674 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/54e09508-bf0d-4f03-b35e-69b4b61d3a9f-host-var-lib-cni-bin\") pod \"multus-5dt8f\" (UID: \"54e09508-bf0d-4f03-b35e-69b4b61d3a9f\") " pod="openshift-multus/multus-5dt8f" Sep 18 06:22:22.407017 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.406715 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/54e09508-bf0d-4f03-b35e-69b4b61d3a9f-host-var-lib-kubelet\") pod \"multus-5dt8f\" (UID: \"54e09508-bf0d-4f03-b35e-69b4b61d3a9f\") " pod="openshift-multus/multus-5dt8f" Sep 18 06:22:22.407017 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.406716 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/27cccfd2-ff8e-4f71-bc24-5072f09b7f21-host\") pod \"node-ca-xccgg\" (UID: \"27cccfd2-ff8e-4f71-bc24-5072f09b7f21\") " pod="openshift-image-registry/node-ca-xccgg" Sep 18 06:22:22.407017 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.406678 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/27cccfd2-ff8e-4f71-bc24-5072f09b7f21-host\") pod \"node-ca-xccgg\" (UID: \"27cccfd2-ff8e-4f71-bc24-5072f09b7f21\") " pod="openshift-image-registry/node-ca-xccgg" Sep 18 06:22:22.407723 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.406761 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12-lib-modules\") pod \"tuned-w7cmt\" (UID: \"e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12\") " pod="openshift-cluster-node-tuning-operator/tuned-w7cmt" Sep 18 06:22:22.407723 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.406785 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12-etc-tuned\") pod \"tuned-w7cmt\" (UID: \"e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12\") " pod="openshift-cluster-node-tuning-operator/tuned-w7cmt" Sep 18 06:22:22.407723 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.406810 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0b83c068-4913-4ee5-85ed-17f44dc9f0ca-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q4gmz\" (UID: \"0b83c068-4913-4ee5-85ed-17f44dc9f0ca\") " pod="openshift-multus/multus-additional-cni-plugins-q4gmz" Sep 18 06:22:22.407723 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.406837 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m7w4\" (UniqueName: \"kubernetes.io/projected/0b83c068-4913-4ee5-85ed-17f44dc9f0ca-kube-api-access-8m7w4\") pod \"multus-additional-cni-plugins-q4gmz\" (UID: \"0b83c068-4913-4ee5-85ed-17f44dc9f0ca\") " pod="openshift-multus/multus-additional-cni-plugins-q4gmz" Sep 18 06:22:22.407723 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.406861 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e57e7a14-2ca6-4f41-875f-822739fa1c43-original-pull-secret\") pod \"global-pull-secret-syncer-2jzcv\" (UID: \"e57e7a14-2ca6-4f41-875f-822739fa1c43\") " pod="kube-system/global-pull-secret-syncer-2jzcv" Sep 18 06:22:22.407723 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.406884 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9c2fc9d9-19db-4703-aa25-95b5f5a42e99-iptables-alerter-script\") pod \"iptables-alerter-p94t6\" (UID: \"9c2fc9d9-19db-4703-aa25-95b5f5a42e99\") " pod="openshift-network-operator/iptables-alerter-p94t6" Sep 18 06:22:22.407723 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.406906 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/54e09508-bf0d-4f03-b35e-69b4b61d3a9f-multus-socket-dir-parent\") pod \"multus-5dt8f\" (UID: \"54e09508-bf0d-4f03-b35e-69b4b61d3a9f\") " pod="openshift-multus/multus-5dt8f" Sep 18 06:22:22.407723 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.406939 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/54e09508-bf0d-4f03-b35e-69b4b61d3a9f-host-run-multus-certs\") pod \"multus-5dt8f\" (UID: \"54e09508-bf0d-4f03-b35e-69b4b61d3a9f\") " pod="openshift-multus/multus-5dt8f" Sep 18 06:22:22.407723 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.406966 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12-etc-sysctl-conf\") pod \"tuned-w7cmt\" (UID: \"e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12\") " pod="openshift-cluster-node-tuning-operator/tuned-w7cmt" Sep 18 06:22:22.407723 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.406990 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0b83c068-4913-4ee5-85ed-17f44dc9f0ca-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q4gmz\" (UID: \"0b83c068-4913-4ee5-85ed-17f44dc9f0ca\") " pod="openshift-multus/multus-additional-cni-plugins-q4gmz" Sep 18 06:22:22.407723 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.407012 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/54e09508-bf0d-4f03-b35e-69b4b61d3a9f-multus-socket-dir-parent\") pod \"multus-5dt8f\" (UID: \"54e09508-bf0d-4f03-b35e-69b4b61d3a9f\") " pod="openshift-multus/multus-5dt8f" Sep 18 06:22:22.407723 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.407021 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e57e7a14-2ca6-4f41-875f-822739fa1c43-kubelet-config\") pod \"global-pull-secret-syncer-2jzcv\" (UID: \"e57e7a14-2ca6-4f41-875f-822739fa1c43\") " pod="kube-system/global-pull-secret-syncer-2jzcv" Sep 18 06:22:22.407723 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.407021 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/54e09508-bf0d-4f03-b35e-69b4b61d3a9f-host-run-multus-certs\") pod \"multus-5dt8f\" (UID: \"54e09508-bf0d-4f03-b35e-69b4b61d3a9f\") " pod="openshift-multus/multus-5dt8f" Sep 18 06:22:22.407723 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.407057 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c7e27a94-0a48-466c-abdb-9f6b0758e358-hosts-file\") pod \"node-resolver-kgx9w\" (UID: \"c7e27a94-0a48-466c-abdb-9f6b0758e358\") " pod="openshift-dns/node-resolver-kgx9w" Sep 18 06:22:22.407723 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.407079 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4d3101c6-973c-48d6-8efe-f4fab4282824-etc-selinux\") pod \"aws-ebs-csi-driver-node-wmmq2\" (UID: \"4d3101c6-973c-48d6-8efe-f4fab4282824\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmmq2" Sep 18 06:22:22.407723 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.407095 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb059951-e263-40da-9155-4fd0cf242820-host-run-ovn-kubernetes\") pod \"ovnkube-node-hjr4h\" (UID: \"eb059951-e263-40da-9155-4fd0cf242820\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:22.408352 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.407123 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12-etc-sysconfig\") pod \"tuned-w7cmt\" (UID: \"e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12\") " pod="openshift-cluster-node-tuning-operator/tuned-w7cmt" Sep 18 06:22:22.408352 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.407152 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb059951-e263-40da-9155-4fd0cf242820-run-openvswitch\") pod \"ovnkube-node-hjr4h\" (UID: \"eb059951-e263-40da-9155-4fd0cf242820\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:22.408352 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.407176 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb059951-e263-40da-9155-4fd0cf242820-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hjr4h\" (UID: \"eb059951-e263-40da-9155-4fd0cf242820\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:22.408352 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.407203 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/54e09508-bf0d-4f03-b35e-69b4b61d3a9f-cni-binary-copy\") pod \"multus-5dt8f\" (UID: \"54e09508-bf0d-4f03-b35e-69b4b61d3a9f\") " pod="openshift-multus/multus-5dt8f" Sep 18 06:22:22.408352 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.407227 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12-etc-modprobe-d\") pod \"tuned-w7cmt\" (UID: \"e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12\") " pod="openshift-cluster-node-tuning-operator/tuned-w7cmt" Sep 18 06:22:22.408352 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.407248 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12-host\") pod \"tuned-w7cmt\" (UID: \"e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12\") " pod="openshift-cluster-node-tuning-operator/tuned-w7cmt" Sep 18 06:22:22.408352 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.407270 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4d3101c6-973c-48d6-8efe-f4fab4282824-registration-dir\") pod \"aws-ebs-csi-driver-node-wmmq2\" (UID: \"4d3101c6-973c-48d6-8efe-f4fab4282824\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmmq2" Sep 18 06:22:22.408352 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.407305 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvsvc\" (UniqueName: \"kubernetes.io/projected/27cccfd2-ff8e-4f71-bc24-5072f09b7f21-kube-api-access-lvsvc\") pod \"node-ca-xccgg\" (UID: \"27cccfd2-ff8e-4f71-bc24-5072f09b7f21\") " pod="openshift-image-registry/node-ca-xccgg" Sep 18 06:22:22.408352 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.407336 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12-var-lib-kubelet\") pod \"tuned-w7cmt\" (UID: \"e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12\") " pod="openshift-cluster-node-tuning-operator/tuned-w7cmt" Sep 18 06:22:22.408352 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.407358 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb059951-e263-40da-9155-4fd0cf242820-host-run-netns\") pod \"ovnkube-node-hjr4h\" (UID: \"eb059951-e263-40da-9155-4fd0cf242820\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:22.408352 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.407397 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/54e09508-bf0d-4f03-b35e-69b4b61d3a9f-multus-daemon-config\") pod \"multus-5dt8f\" (UID: \"54e09508-bf0d-4f03-b35e-69b4b61d3a9f\") " pod="openshift-multus/multus-5dt8f" Sep 18 06:22:22.408352 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.407471 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eb059951-e263-40da-9155-4fd0cf242820-env-overrides\") pod \"ovnkube-node-hjr4h\" (UID: \"eb059951-e263-40da-9155-4fd0cf242820\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:22.408352 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.407510 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/27cccfd2-ff8e-4f71-bc24-5072f09b7f21-serviceca\") pod \"node-ca-xccgg\" (UID: \"27cccfd2-ff8e-4f71-bc24-5072f09b7f21\") " pod="openshift-image-registry/node-ca-xccgg" Sep 18 06:22:22.408352 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.407560 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/54e09508-bf0d-4f03-b35e-69b4b61d3a9f-etc-kubernetes\") pod \"multus-5dt8f\" (UID: \"54e09508-bf0d-4f03-b35e-69b4b61d3a9f\") " pod="openshift-multus/multus-5dt8f" Sep 18 06:22:22.408352 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.407678 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12-etc-kubernetes\") pod \"tuned-w7cmt\" (UID: \"e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12\") " pod="openshift-cluster-node-tuning-operator/tuned-w7cmt" Sep 18 06:22:22.408352 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.407690 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/54e09508-bf0d-4f03-b35e-69b4b61d3a9f-etc-kubernetes\") pod \"multus-5dt8f\" (UID: \"54e09508-bf0d-4f03-b35e-69b4b61d3a9f\") " pod="openshift-multus/multus-5dt8f" Sep 18 06:22:22.408352 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.407692 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/54e09508-bf0d-4f03-b35e-69b4b61d3a9f-cni-binary-copy\") pod \"multus-5dt8f\" (UID: \"54e09508-bf0d-4f03-b35e-69b4b61d3a9f\") " pod="openshift-multus/multus-5dt8f" Sep 18 06:22:22.408910 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.407718 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12-etc-sysctl-d\") pod \"tuned-w7cmt\" (UID: \"e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12\") " pod="openshift-cluster-node-tuning-operator/tuned-w7cmt" Sep 18 06:22:22.408910 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.407742 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eb059951-e263-40da-9155-4fd0cf242820-node-log\") pod \"ovnkube-node-hjr4h\" (UID: \"eb059951-e263-40da-9155-4fd0cf242820\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:22.408910 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.407766 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eb059951-e263-40da-9155-4fd0cf242820-host-cni-netd\") pod \"ovnkube-node-hjr4h\" (UID: \"eb059951-e263-40da-9155-4fd0cf242820\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:22.408910 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.407790 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eb059951-e263-40da-9155-4fd0cf242820-ovnkube-script-lib\") pod \"ovnkube-node-hjr4h\" (UID: \"eb059951-e263-40da-9155-4fd0cf242820\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:22.408910 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.407812 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12-etc-systemd\") pod \"tuned-w7cmt\" (UID: \"e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12\") " pod="openshift-cluster-node-tuning-operator/tuned-w7cmt" Sep 18 06:22:22.408910 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.407835 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srldb\" (UniqueName: \"kubernetes.io/projected/c7e27a94-0a48-466c-abdb-9f6b0758e358-kube-api-access-srldb\") pod \"node-resolver-kgx9w\" (UID: \"c7e27a94-0a48-466c-abdb-9f6b0758e358\") " pod="openshift-dns/node-resolver-kgx9w" Sep 18 06:22:22.408910 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.407860 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slppz\" (UniqueName: \"kubernetes.io/projected/54e09508-bf0d-4f03-b35e-69b4b61d3a9f-kube-api-access-slppz\") pod \"multus-5dt8f\" (UID: \"54e09508-bf0d-4f03-b35e-69b4b61d3a9f\") " pod="openshift-multus/multus-5dt8f" Sep 18 06:22:22.408910 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.407884 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/631676f2-32af-4b0d-834b-41c7c59db5ba-agent-certs\") pod \"konnectivity-agent-sxrxx\" (UID: \"631676f2-32af-4b0d-834b-41c7c59db5ba\") " pod="kube-system/konnectivity-agent-sxrxx" Sep 18 06:22:22.408910 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.407935 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/27cccfd2-ff8e-4f71-bc24-5072f09b7f21-serviceca\") pod \"node-ca-xccgg\" (UID: \"27cccfd2-ff8e-4f71-bc24-5072f09b7f21\") " pod="openshift-image-registry/node-ca-xccgg" Sep 18 06:22:22.408910 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.407936 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cfng\" (UniqueName: \"kubernetes.io/projected/e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12-kube-api-access-6cfng\") pod \"tuned-w7cmt\" (UID: \"e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12\") " pod="openshift-cluster-node-tuning-operator/tuned-w7cmt" Sep 18 06:22:22.408910 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.407973 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0b83c068-4913-4ee5-85ed-17f44dc9f0ca-system-cni-dir\") pod \"multus-additional-cni-plugins-q4gmz\" (UID: \"0b83c068-4913-4ee5-85ed-17f44dc9f0ca\") " pod="openshift-multus/multus-additional-cni-plugins-q4gmz" Sep 18 06:22:22.408910 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.407996 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0b83c068-4913-4ee5-85ed-17f44dc9f0ca-cni-binary-copy\") pod \"multus-additional-cni-plugins-q4gmz\" (UID: \"0b83c068-4913-4ee5-85ed-17f44dc9f0ca\") " pod="openshift-multus/multus-additional-cni-plugins-q4gmz" Sep 18 06:22:22.408910 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.408018 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eb059951-e263-40da-9155-4fd0cf242820-log-socket\") pod \"ovnkube-node-hjr4h\" (UID: \"eb059951-e263-40da-9155-4fd0cf242820\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:22.408910 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.408040 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eb059951-e263-40da-9155-4fd0cf242820-ovnkube-config\") pod \"ovnkube-node-hjr4h\" (UID: \"eb059951-e263-40da-9155-4fd0cf242820\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:22.412949 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:22.412925 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 18 06:22:22.412949 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:22.412947 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 18 06:22:22.413085 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:22.412957 2567 projected.go:196] Error preparing data for projected volume kube-api-access-pwmtn for pod openshift-network-diagnostics/network-check-target-5rhpx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 06:22:22.413085 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:22.413026 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce5f0b70-0633-4019-a805-85d798616ac2-kube-api-access-pwmtn podName:ce5f0b70-0633-4019-a805-85d798616ac2 nodeName:}" failed. No retries permitted until 2026-09-18 06:22:22.913007316 +0000 UTC m=+4.652423503 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-pwmtn" (UniqueName: "kubernetes.io/projected/ce5f0b70-0633-4019-a805-85d798616ac2-kube-api-access-pwmtn") pod "network-check-target-5rhpx" (UID: "ce5f0b70-0633-4019-a805-85d798616ac2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 06:22:22.416768 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.416728 2567 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 18 06:22:22.420420 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.420401 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvsvc\" (UniqueName: \"kubernetes.io/projected/27cccfd2-ff8e-4f71-bc24-5072f09b7f21-kube-api-access-lvsvc\") pod \"node-ca-xccgg\" (UID: \"27cccfd2-ff8e-4f71-bc24-5072f09b7f21\") " pod="openshift-image-registry/node-ca-xccgg" Sep 18 06:22:22.420503 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.420422 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7b74\" (UniqueName: \"kubernetes.io/projected/9c2fc9d9-19db-4703-aa25-95b5f5a42e99-kube-api-access-l7b74\") pod \"iptables-alerter-p94t6\" (UID: \"9c2fc9d9-19db-4703-aa25-95b5f5a42e99\") " pod="openshift-network-operator/iptables-alerter-p94t6" Sep 18 06:22:22.420541 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.420499 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slppz\" (UniqueName: \"kubernetes.io/projected/54e09508-bf0d-4f03-b35e-69b4b61d3a9f-kube-api-access-slppz\") pod \"multus-5dt8f\" (UID: \"54e09508-bf0d-4f03-b35e-69b4b61d3a9f\") " pod="openshift-multus/multus-5dt8f" Sep 18 06:22:22.447060 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.446927 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-104.ec2.internal" event={"ID":"093f6c5a5a5dfa9d45afb8587676bdd0","Type":"ContainerStarted","Data":"e36caf5ed2911b18b8413f60581995f10fbaab2a0eec8605de7ac361c609e3b6"} Sep 18 06:22:22.448661 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.448637 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-104.ec2.internal" event={"ID":"f23d030455f844c3f1c0ef497274c121","Type":"ContainerStarted","Data":"64127663be7c8084c31c8f475955aeae39346f3306e45f44bdf59848162755bf"} Sep 18 06:22:22.509303 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.509279 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66dca255-1289-49ec-b200-e950a30e3d9c-metrics-certs\") pod \"network-metrics-daemon-k77fk\" (UID: \"66dca255-1289-49ec-b200-e950a30e3d9c\") " pod="openshift-multus/network-metrics-daemon-k77fk" Sep 18 06:22:22.509303 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.509306 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb059951-e263-40da-9155-4fd0cf242820-host-cni-bin\") pod \"ovnkube-node-hjr4h\" (UID: \"eb059951-e263-40da-9155-4fd0cf242820\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:22.509492 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.509322 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/631676f2-32af-4b0d-834b-41c7c59db5ba-konnectivity-ca\") pod \"konnectivity-agent-sxrxx\" (UID: \"631676f2-32af-4b0d-834b-41c7c59db5ba\") " pod="kube-system/konnectivity-agent-sxrxx" Sep 18 06:22:22.509492 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.509345 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb059951-e263-40da-9155-4fd0cf242820-host-slash\") pod \"ovnkube-node-hjr4h\" (UID: \"eb059951-e263-40da-9155-4fd0cf242820\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:22.509492 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.509385 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eb059951-e263-40da-9155-4fd0cf242820-run-systemd\") pod \"ovnkube-node-hjr4h\" (UID: \"eb059951-e263-40da-9155-4fd0cf242820\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:22.509492 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.509403 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb059951-e263-40da-9155-4fd0cf242820-host-cni-bin\") pod \"ovnkube-node-hjr4h\" (UID: \"eb059951-e263-40da-9155-4fd0cf242820\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:22.509492 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.509413 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e57e7a14-2ca6-4f41-875f-822739fa1c43-dbus\") pod \"global-pull-secret-syncer-2jzcv\" (UID: \"e57e7a14-2ca6-4f41-875f-822739fa1c43\") " pod="kube-system/global-pull-secret-syncer-2jzcv" Sep 18 06:22:22.509492 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:22.509450 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 06:22:22.509492 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.509463 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eb059951-e263-40da-9155-4fd0cf242820-run-systemd\") pod \"ovnkube-node-hjr4h\" (UID: \"eb059951-e263-40da-9155-4fd0cf242820\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:22.509492 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.509470 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb059951-e263-40da-9155-4fd0cf242820-host-slash\") pod \"ovnkube-node-hjr4h\" (UID: \"eb059951-e263-40da-9155-4fd0cf242820\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:22.509492 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.509493 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eb059951-e263-40da-9155-4fd0cf242820-ovn-node-metrics-cert\") pod \"ovnkube-node-hjr4h\" (UID: \"eb059951-e263-40da-9155-4fd0cf242820\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:22.509855 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.509497 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e57e7a14-2ca6-4f41-875f-822739fa1c43-dbus\") pod \"global-pull-secret-syncer-2jzcv\" (UID: \"e57e7a14-2ca6-4f41-875f-822739fa1c43\") " pod="kube-system/global-pull-secret-syncer-2jzcv" Sep 18 06:22:22.509855 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:22.509522 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66dca255-1289-49ec-b200-e950a30e3d9c-metrics-certs podName:66dca255-1289-49ec-b200-e950a30e3d9c nodeName:}" failed. No retries permitted until 2026-09-18 06:22:23.009499558 +0000 UTC m=+4.748915712 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66dca255-1289-49ec-b200-e950a30e3d9c-metrics-certs") pod "network-metrics-daemon-k77fk" (UID: "66dca255-1289-49ec-b200-e950a30e3d9c") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 06:22:22.509855 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.509564 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4d3101c6-973c-48d6-8efe-f4fab4282824-sys-fs\") pod \"aws-ebs-csi-driver-node-wmmq2\" (UID: \"4d3101c6-973c-48d6-8efe-f4fab4282824\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmmq2" Sep 18 06:22:22.509855 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.509591 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sct85\" (UniqueName: \"kubernetes.io/projected/66dca255-1289-49ec-b200-e950a30e3d9c-kube-api-access-sct85\") pod \"network-metrics-daemon-k77fk\" (UID: \"66dca255-1289-49ec-b200-e950a30e3d9c\") " pod="openshift-multus/network-metrics-daemon-k77fk" Sep 18 06:22:22.509855 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.509617 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb059951-e263-40da-9155-4fd0cf242820-var-lib-openvswitch\") pod \"ovnkube-node-hjr4h\" (UID: \"eb059951-e263-40da-9155-4fd0cf242820\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:22.509855 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.509623 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4d3101c6-973c-48d6-8efe-f4fab4282824-sys-fs\") pod \"aws-ebs-csi-driver-node-wmmq2\" (UID: \"4d3101c6-973c-48d6-8efe-f4fab4282824\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmmq2" Sep 18 06:22:22.509855 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.509641 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12-sys\") pod \"tuned-w7cmt\" (UID: \"e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12\") " pod="openshift-cluster-node-tuning-operator/tuned-w7cmt" Sep 18 06:22:22.509855 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.509676 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb059951-e263-40da-9155-4fd0cf242820-var-lib-openvswitch\") pod \"ovnkube-node-hjr4h\" (UID: \"eb059951-e263-40da-9155-4fd0cf242820\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:22.509855 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.509688 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12-sys\") pod \"tuned-w7cmt\" (UID: \"e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12\") " pod="openshift-cluster-node-tuning-operator/tuned-w7cmt" Sep 18 06:22:22.509855 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.509795 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12-tmp\") pod \"tuned-w7cmt\" (UID: \"e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12\") " pod="openshift-cluster-node-tuning-operator/tuned-w7cmt" Sep 18 06:22:22.509855 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.509840 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0b83c068-4913-4ee5-85ed-17f44dc9f0ca-cnibin\") pod \"multus-additional-cni-plugins-q4gmz\" (UID: \"0b83c068-4913-4ee5-85ed-17f44dc9f0ca\") " pod="openshift-multus/multus-additional-cni-plugins-q4gmz" Sep 18 06:22:22.510361 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.509867 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4d3101c6-973c-48d6-8efe-f4fab4282824-device-dir\") pod \"aws-ebs-csi-driver-node-wmmq2\" (UID: \"4d3101c6-973c-48d6-8efe-f4fab4282824\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmmq2" Sep 18 06:22:22.510361 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.509892 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eb059951-e263-40da-9155-4fd0cf242820-systemd-units\") pod \"ovnkube-node-hjr4h\" (UID: \"eb059951-e263-40da-9155-4fd0cf242820\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:22.510361 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.509905 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0b83c068-4913-4ee5-85ed-17f44dc9f0ca-cnibin\") pod \"multus-additional-cni-plugins-q4gmz\" (UID: \"0b83c068-4913-4ee5-85ed-17f44dc9f0ca\") " pod="openshift-multus/multus-additional-cni-plugins-q4gmz" Sep 18 06:22:22.510361 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.509919 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12-lib-modules\") pod \"tuned-w7cmt\" (UID: \"e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12\") " pod="openshift-cluster-node-tuning-operator/tuned-w7cmt" Sep 18 06:22:22.510361 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.509951 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12-etc-tuned\") pod \"tuned-w7cmt\" (UID: \"e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12\") " pod="openshift-cluster-node-tuning-operator/tuned-w7cmt" Sep 18 06:22:22.510361 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.509950 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/631676f2-32af-4b0d-834b-41c7c59db5ba-konnectivity-ca\") pod \"konnectivity-agent-sxrxx\" (UID: \"631676f2-32af-4b0d-834b-41c7c59db5ba\") " pod="kube-system/konnectivity-agent-sxrxx" Sep 18 06:22:22.510361 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.509978 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0b83c068-4913-4ee5-85ed-17f44dc9f0ca-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q4gmz\" (UID: \"0b83c068-4913-4ee5-85ed-17f44dc9f0ca\") " pod="openshift-multus/multus-additional-cni-plugins-q4gmz" Sep 18 06:22:22.510361 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.509988 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4d3101c6-973c-48d6-8efe-f4fab4282824-device-dir\") pod \"aws-ebs-csi-driver-node-wmmq2\" (UID: \"4d3101c6-973c-48d6-8efe-f4fab4282824\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmmq2" Sep 18 06:22:22.510361 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.510005 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m7w4\" (UniqueName: \"kubernetes.io/projected/0b83c068-4913-4ee5-85ed-17f44dc9f0ca-kube-api-access-8m7w4\") pod \"multus-additional-cni-plugins-q4gmz\" (UID: \"0b83c068-4913-4ee5-85ed-17f44dc9f0ca\") " pod="openshift-multus/multus-additional-cni-plugins-q4gmz" Sep 18 06:22:22.510361 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.510031 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e57e7a14-2ca6-4f41-875f-822739fa1c43-original-pull-secret\") pod \"global-pull-secret-syncer-2jzcv\" (UID: \"e57e7a14-2ca6-4f41-875f-822739fa1c43\") " pod="kube-system/global-pull-secret-syncer-2jzcv" Sep 18 06:22:22.510361 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.510006 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eb059951-e263-40da-9155-4fd0cf242820-systemd-units\") pod \"ovnkube-node-hjr4h\" (UID: \"eb059951-e263-40da-9155-4fd0cf242820\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:22.510361 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.510060 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12-etc-sysctl-conf\") pod \"tuned-w7cmt\" (UID: \"e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12\") " pod="openshift-cluster-node-tuning-operator/tuned-w7cmt" Sep 18 06:22:22.510361 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.510083 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0b83c068-4913-4ee5-85ed-17f44dc9f0ca-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q4gmz\" (UID: \"0b83c068-4913-4ee5-85ed-17f44dc9f0ca\") " pod="openshift-multus/multus-additional-cni-plugins-q4gmz" Sep 18 06:22:22.510361 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.510106 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e57e7a14-2ca6-4f41-875f-822739fa1c43-kubelet-config\") pod \"global-pull-secret-syncer-2jzcv\" (UID: \"e57e7a14-2ca6-4f41-875f-822739fa1c43\") " pod="kube-system/global-pull-secret-syncer-2jzcv" Sep 18 06:22:22.510361 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.510127 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c7e27a94-0a48-466c-abdb-9f6b0758e358-hosts-file\") pod \"node-resolver-kgx9w\" (UID: \"c7e27a94-0a48-466c-abdb-9f6b0758e358\") " pod="openshift-dns/node-resolver-kgx9w" Sep 18 06:22:22.510361 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.510156 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4d3101c6-973c-48d6-8efe-f4fab4282824-etc-selinux\") pod \"aws-ebs-csi-driver-node-wmmq2\" (UID: \"4d3101c6-973c-48d6-8efe-f4fab4282824\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmmq2" Sep 18 06:22:22.510361 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.510172 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb059951-e263-40da-9155-4fd0cf242820-host-run-ovn-kubernetes\") pod \"ovnkube-node-hjr4h\" (UID: \"eb059951-e263-40da-9155-4fd0cf242820\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:22.511202 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.510186 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12-etc-sysconfig\") pod \"tuned-w7cmt\" (UID: \"e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12\") " pod="openshift-cluster-node-tuning-operator/tuned-w7cmt" Sep 18 06:22:22.511202 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.510201 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb059951-e263-40da-9155-4fd0cf242820-run-openvswitch\") pod \"ovnkube-node-hjr4h\" (UID: \"eb059951-e263-40da-9155-4fd0cf242820\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:22.511202 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.510229 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb059951-e263-40da-9155-4fd0cf242820-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hjr4h\" (UID: \"eb059951-e263-40da-9155-4fd0cf242820\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:22.511202 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.510266 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12-etc-modprobe-d\") pod \"tuned-w7cmt\" (UID: \"e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12\") " pod="openshift-cluster-node-tuning-operator/tuned-w7cmt" Sep 18 06:22:22.511202 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.510288 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12-host\") pod \"tuned-w7cmt\" (UID: \"e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12\") " pod="openshift-cluster-node-tuning-operator/tuned-w7cmt" Sep 18 06:22:22.511202 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.510309 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4d3101c6-973c-48d6-8efe-f4fab4282824-registration-dir\") pod \"aws-ebs-csi-driver-node-wmmq2\" (UID: \"4d3101c6-973c-48d6-8efe-f4fab4282824\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmmq2" Sep 18 06:22:22.511202 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:22.510340 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 18 06:22:22.511202 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.510405 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12-var-lib-kubelet\") pod \"tuned-w7cmt\" (UID: \"e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12\") " pod="openshift-cluster-node-tuning-operator/tuned-w7cmt" Sep 18 06:22:22.511202 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:22.510463 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e57e7a14-2ca6-4f41-875f-822739fa1c43-original-pull-secret podName:e57e7a14-2ca6-4f41-875f-822739fa1c43 nodeName:}" failed. No retries permitted until 2026-09-18 06:22:23.010444646 +0000 UTC m=+4.749860800 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e57e7a14-2ca6-4f41-875f-822739fa1c43-original-pull-secret") pod "global-pull-secret-syncer-2jzcv" (UID: "e57e7a14-2ca6-4f41-875f-822739fa1c43") : object "kube-system"/"original-pull-secret" not registered Sep 18 06:22:22.511202 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.510033 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12-lib-modules\") pod \"tuned-w7cmt\" (UID: \"e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12\") " pod="openshift-cluster-node-tuning-operator/tuned-w7cmt" Sep 18 06:22:22.511202 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.510512 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4d3101c6-973c-48d6-8efe-f4fab4282824-etc-selinux\") pod \"aws-ebs-csi-driver-node-wmmq2\" (UID: \"4d3101c6-973c-48d6-8efe-f4fab4282824\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmmq2" Sep 18 06:22:22.511202 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.510561 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb059951-e263-40da-9155-4fd0cf242820-host-run-ovn-kubernetes\") pod \"ovnkube-node-hjr4h\" (UID: \"eb059951-e263-40da-9155-4fd0cf242820\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:22.511202 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.510583 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0b83c068-4913-4ee5-85ed-17f44dc9f0ca-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q4gmz\" (UID: \"0b83c068-4913-4ee5-85ed-17f44dc9f0ca\") " pod="openshift-multus/multus-additional-cni-plugins-q4gmz" Sep 18 06:22:22.511202 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.510611 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12-etc-sysconfig\") pod \"tuned-w7cmt\" (UID: \"e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12\") " pod="openshift-cluster-node-tuning-operator/tuned-w7cmt" Sep 18 06:22:22.511202 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.510615 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12-etc-sysctl-conf\") pod \"tuned-w7cmt\" (UID: \"e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12\") " pod="openshift-cluster-node-tuning-operator/tuned-w7cmt" Sep 18 06:22:22.511202 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.510334 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12-var-lib-kubelet\") pod \"tuned-w7cmt\" (UID: \"e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12\") " pod="openshift-cluster-node-tuning-operator/tuned-w7cmt" Sep 18 06:22:22.511202 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.510664 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12-host\") pod \"tuned-w7cmt\" (UID: \"e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12\") " pod="openshift-cluster-node-tuning-operator/tuned-w7cmt" Sep 18 06:22:22.512018 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.510665 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb059951-e263-40da-9155-4fd0cf242820-host-run-netns\") pod \"ovnkube-node-hjr4h\" (UID: \"eb059951-e263-40da-9155-4fd0cf242820\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:22.512018 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.510696 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eb059951-e263-40da-9155-4fd0cf242820-env-overrides\") pod \"ovnkube-node-hjr4h\" (UID: \"eb059951-e263-40da-9155-4fd0cf242820\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:22.512018 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.510703 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb059951-e263-40da-9155-4fd0cf242820-run-openvswitch\") pod \"ovnkube-node-hjr4h\" (UID: \"eb059951-e263-40da-9155-4fd0cf242820\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:22.512018 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.510722 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12-etc-kubernetes\") pod \"tuned-w7cmt\" (UID: \"e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12\") " pod="openshift-cluster-node-tuning-operator/tuned-w7cmt" Sep 18 06:22:22.512018 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.510749 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12-etc-sysctl-d\") pod \"tuned-w7cmt\" (UID: \"e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12\") " pod="openshift-cluster-node-tuning-operator/tuned-w7cmt" Sep 18 06:22:22.512018 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.510753 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e57e7a14-2ca6-4f41-875f-822739fa1c43-kubelet-config\") pod \"global-pull-secret-syncer-2jzcv\" (UID: \"e57e7a14-2ca6-4f41-875f-822739fa1c43\") " pod="kube-system/global-pull-secret-syncer-2jzcv" Sep 18 06:22:22.512018 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.510751 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12-etc-modprobe-d\") pod \"tuned-w7cmt\" (UID: \"e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12\") " pod="openshift-cluster-node-tuning-operator/tuned-w7cmt" Sep 18 06:22:22.512018 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.510810 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c7e27a94-0a48-466c-abdb-9f6b0758e358-hosts-file\") pod \"node-resolver-kgx9w\" (UID: \"c7e27a94-0a48-466c-abdb-9f6b0758e358\") " pod="openshift-dns/node-resolver-kgx9w" Sep 18 06:22:22.512018 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.510811 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4d3101c6-973c-48d6-8efe-f4fab4282824-registration-dir\") pod \"aws-ebs-csi-driver-node-wmmq2\" (UID: \"4d3101c6-973c-48d6-8efe-f4fab4282824\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmmq2" Sep 18 06:22:22.512018 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.510832 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eb059951-e263-40da-9155-4fd0cf242820-node-log\") pod \"ovnkube-node-hjr4h\" (UID: \"eb059951-e263-40da-9155-4fd0cf242820\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:22.512018 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.510861 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12-etc-kubernetes\") pod \"tuned-w7cmt\" (UID: \"e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12\") " pod="openshift-cluster-node-tuning-operator/tuned-w7cmt" Sep 18 06:22:22.512018 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.510868 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eb059951-e263-40da-9155-4fd0cf242820-host-cni-netd\") pod \"ovnkube-node-hjr4h\" (UID: \"eb059951-e263-40da-9155-4fd0cf242820\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:22.512018 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.510893 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eb059951-e263-40da-9155-4fd0cf242820-ovnkube-script-lib\") pod \"ovnkube-node-hjr4h\" (UID: \"eb059951-e263-40da-9155-4fd0cf242820\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:22.512018 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.510918 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12-etc-systemd\") pod \"tuned-w7cmt\" (UID: \"e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12\") " pod="openshift-cluster-node-tuning-operator/tuned-w7cmt" Sep 18 06:22:22.512018 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.510945 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srldb\" (UniqueName: \"kubernetes.io/projected/c7e27a94-0a48-466c-abdb-9f6b0758e358-kube-api-access-srldb\") pod \"node-resolver-kgx9w\" (UID: \"c7e27a94-0a48-466c-abdb-9f6b0758e358\") " pod="openshift-dns/node-resolver-kgx9w" Sep 18 06:22:22.512018 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.510963 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12-etc-sysctl-d\") pod \"tuned-w7cmt\" (UID: \"e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12\") " pod="openshift-cluster-node-tuning-operator/tuned-w7cmt" Sep 18 06:22:22.512018 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.510969 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/631676f2-32af-4b0d-834b-41c7c59db5ba-agent-certs\") pod \"konnectivity-agent-sxrxx\" (UID: \"631676f2-32af-4b0d-834b-41c7c59db5ba\") " pod="kube-system/konnectivity-agent-sxrxx" Sep 18 06:22:22.512847 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.510994 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cfng\" (UniqueName: \"kubernetes.io/projected/e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12-kube-api-access-6cfng\") pod \"tuned-w7cmt\" (UID: \"e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12\") " pod="openshift-cluster-node-tuning-operator/tuned-w7cmt" Sep 18 06:22:22.512847 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.511003 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eb059951-e263-40da-9155-4fd0cf242820-host-cni-netd\") pod \"ovnkube-node-hjr4h\" (UID: \"eb059951-e263-40da-9155-4fd0cf242820\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:22.512847 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.511018 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0b83c068-4913-4ee5-85ed-17f44dc9f0ca-system-cni-dir\") pod \"multus-additional-cni-plugins-q4gmz\" (UID: \"0b83c068-4913-4ee5-85ed-17f44dc9f0ca\") " pod="openshift-multus/multus-additional-cni-plugins-q4gmz" Sep 18 06:22:22.512847 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.511043 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0b83c068-4913-4ee5-85ed-17f44dc9f0ca-cni-binary-copy\") pod \"multus-additional-cni-plugins-q4gmz\" (UID: \"0b83c068-4913-4ee5-85ed-17f44dc9f0ca\") " pod="openshift-multus/multus-additional-cni-plugins-q4gmz" Sep 18 06:22:22.512847 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.511065 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eb059951-e263-40da-9155-4fd0cf242820-log-socket\") pod \"ovnkube-node-hjr4h\" (UID: \"eb059951-e263-40da-9155-4fd0cf242820\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:22.512847 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.511074 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb059951-e263-40da-9155-4fd0cf242820-host-run-netns\") pod \"ovnkube-node-hjr4h\" (UID: \"eb059951-e263-40da-9155-4fd0cf242820\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:22.512847 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.511089 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eb059951-e263-40da-9155-4fd0cf242820-ovnkube-config\") pod \"ovnkube-node-hjr4h\" (UID: \"eb059951-e263-40da-9155-4fd0cf242820\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:22.512847 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.511115 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62ffb\" (UniqueName: \"kubernetes.io/projected/4d3101c6-973c-48d6-8efe-f4fab4282824-kube-api-access-62ffb\") pod \"aws-ebs-csi-driver-node-wmmq2\" (UID: \"4d3101c6-973c-48d6-8efe-f4fab4282824\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmmq2" Sep 18 06:22:22.512847 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.511140 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb059951-e263-40da-9155-4fd0cf242820-host-kubelet\") pod \"ovnkube-node-hjr4h\" (UID: \"eb059951-e263-40da-9155-4fd0cf242820\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:22.512847 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.511148 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eb059951-e263-40da-9155-4fd0cf242820-env-overrides\") pod \"ovnkube-node-hjr4h\" (UID: \"eb059951-e263-40da-9155-4fd0cf242820\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:22.512847 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.511163 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb059951-e263-40da-9155-4fd0cf242820-etc-openvswitch\") pod \"ovnkube-node-hjr4h\" (UID: \"eb059951-e263-40da-9155-4fd0cf242820\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:22.512847 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.511202 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzdct\" (UniqueName: \"kubernetes.io/projected/eb059951-e263-40da-9155-4fd0cf242820-kube-api-access-gzdct\") pod \"ovnkube-node-hjr4h\" (UID: \"eb059951-e263-40da-9155-4fd0cf242820\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:22.512847 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.511233 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12-run\") pod \"tuned-w7cmt\" (UID: \"e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12\") " pod="openshift-cluster-node-tuning-operator/tuned-w7cmt" Sep 18 06:22:22.512847 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.511240 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0b83c068-4913-4ee5-85ed-17f44dc9f0ca-system-cni-dir\") pod \"multus-additional-cni-plugins-q4gmz\" (UID: \"0b83c068-4913-4ee5-85ed-17f44dc9f0ca\") " pod="openshift-multus/multus-additional-cni-plugins-q4gmz" Sep 18 06:22:22.512847 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.511260 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4d3101c6-973c-48d6-8efe-f4fab4282824-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wmmq2\" (UID: \"4d3101c6-973c-48d6-8efe-f4fab4282824\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmmq2" Sep 18 06:22:22.512847 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.511314 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4d3101c6-973c-48d6-8efe-f4fab4282824-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wmmq2\" (UID: \"4d3101c6-973c-48d6-8efe-f4fab4282824\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmmq2" Sep 18 06:22:22.512847 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.510669 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb059951-e263-40da-9155-4fd0cf242820-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hjr4h\" (UID: \"eb059951-e263-40da-9155-4fd0cf242820\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:22.513704 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.511620 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12-etc-systemd\") pod \"tuned-w7cmt\" (UID: \"e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12\") " pod="openshift-cluster-node-tuning-operator/tuned-w7cmt" Sep 18 06:22:22.513704 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.511205 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb059951-e263-40da-9155-4fd0cf242820-etc-openvswitch\") pod \"ovnkube-node-hjr4h\" (UID: \"eb059951-e263-40da-9155-4fd0cf242820\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:22.513704 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.511778 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eb059951-e263-40da-9155-4fd0cf242820-ovnkube-script-lib\") pod \"ovnkube-node-hjr4h\" (UID: \"eb059951-e263-40da-9155-4fd0cf242820\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:22.513704 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.511798 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0b83c068-4913-4ee5-85ed-17f44dc9f0ca-cni-binary-copy\") pod \"multus-additional-cni-plugins-q4gmz\" (UID: \"0b83c068-4913-4ee5-85ed-17f44dc9f0ca\") " pod="openshift-multus/multus-additional-cni-plugins-q4gmz" Sep 18 06:22:22.513704 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.511852 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4d3101c6-973c-48d6-8efe-f4fab4282824-socket-dir\") pod \"aws-ebs-csi-driver-node-wmmq2\" (UID: \"4d3101c6-973c-48d6-8efe-f4fab4282824\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmmq2" Sep 18 06:22:22.513704 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.511874 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12-run\") pod \"tuned-w7cmt\" (UID: \"e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12\") " pod="openshift-cluster-node-tuning-operator/tuned-w7cmt" Sep 18 06:22:22.513704 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.511879 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eb059951-e263-40da-9155-4fd0cf242820-run-ovn\") pod \"ovnkube-node-hjr4h\" (UID: \"eb059951-e263-40da-9155-4fd0cf242820\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:22.513704 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.511921 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eb059951-e263-40da-9155-4fd0cf242820-run-ovn\") pod \"ovnkube-node-hjr4h\" (UID: \"eb059951-e263-40da-9155-4fd0cf242820\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:22.513704 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.511925 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0b83c068-4913-4ee5-85ed-17f44dc9f0ca-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q4gmz\" (UID: \"0b83c068-4913-4ee5-85ed-17f44dc9f0ca\") " pod="openshift-multus/multus-additional-cni-plugins-q4gmz" Sep 18 06:22:22.513704 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.511934 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12-tmp\") pod \"tuned-w7cmt\" (UID: \"e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12\") " pod="openshift-cluster-node-tuning-operator/tuned-w7cmt" Sep 18 06:22:22.513704 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.511949 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7e27a94-0a48-466c-abdb-9f6b0758e358-tmp-dir\") pod \"node-resolver-kgx9w\" (UID: \"c7e27a94-0a48-466c-abdb-9f6b0758e358\") " pod="openshift-dns/node-resolver-kgx9w" Sep 18 06:22:22.513704 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.511044 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eb059951-e263-40da-9155-4fd0cf242820-node-log\") pod \"ovnkube-node-hjr4h\" (UID: \"eb059951-e263-40da-9155-4fd0cf242820\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:22.513704 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.511979 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eb059951-e263-40da-9155-4fd0cf242820-log-socket\") pod \"ovnkube-node-hjr4h\" (UID: \"eb059951-e263-40da-9155-4fd0cf242820\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:22.513704 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.512065 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb059951-e263-40da-9155-4fd0cf242820-host-kubelet\") pod \"ovnkube-node-hjr4h\" (UID: \"eb059951-e263-40da-9155-4fd0cf242820\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:22.513704 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.512180 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eb059951-e263-40da-9155-4fd0cf242820-ovn-node-metrics-cert\") pod \"ovnkube-node-hjr4h\" (UID: \"eb059951-e263-40da-9155-4fd0cf242820\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:22.513704 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.512198 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0b83c068-4913-4ee5-85ed-17f44dc9f0ca-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q4gmz\" (UID: \"0b83c068-4913-4ee5-85ed-17f44dc9f0ca\") " pod="openshift-multus/multus-additional-cni-plugins-q4gmz" Sep 18 06:22:22.513704 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.512213 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4d3101c6-973c-48d6-8efe-f4fab4282824-socket-dir\") pod \"aws-ebs-csi-driver-node-wmmq2\" (UID: \"4d3101c6-973c-48d6-8efe-f4fab4282824\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmmq2" Sep 18 06:22:22.513704 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.512345 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7e27a94-0a48-466c-abdb-9f6b0758e358-tmp-dir\") pod \"node-resolver-kgx9w\" (UID: \"c7e27a94-0a48-466c-abdb-9f6b0758e358\") " pod="openshift-dns/node-resolver-kgx9w" Sep 18 06:22:22.514420 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.512634 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eb059951-e263-40da-9155-4fd0cf242820-ovnkube-config\") pod \"ovnkube-node-hjr4h\" (UID: \"eb059951-e263-40da-9155-4fd0cf242820\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:22.514420 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.512775 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12-etc-tuned\") pod \"tuned-w7cmt\" (UID: \"e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12\") " pod="openshift-cluster-node-tuning-operator/tuned-w7cmt" Sep 18 06:22:22.514420 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.512780 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0b83c068-4913-4ee5-85ed-17f44dc9f0ca-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q4gmz\" (UID: \"0b83c068-4913-4ee5-85ed-17f44dc9f0ca\") " pod="openshift-multus/multus-additional-cni-plugins-q4gmz" Sep 18 06:22:22.514420 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.514359 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/631676f2-32af-4b0d-834b-41c7c59db5ba-agent-certs\") pod \"konnectivity-agent-sxrxx\" (UID: \"631676f2-32af-4b0d-834b-41c7c59db5ba\") " pod="kube-system/konnectivity-agent-sxrxx" Sep 18 06:22:22.519573 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.519529 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sct85\" (UniqueName: \"kubernetes.io/projected/66dca255-1289-49ec-b200-e950a30e3d9c-kube-api-access-sct85\") pod \"network-metrics-daemon-k77fk\" (UID: \"66dca255-1289-49ec-b200-e950a30e3d9c\") " pod="openshift-multus/network-metrics-daemon-k77fk" Sep 18 06:22:22.520949 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.520922 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srldb\" (UniqueName: \"kubernetes.io/projected/c7e27a94-0a48-466c-abdb-9f6b0758e358-kube-api-access-srldb\") pod \"node-resolver-kgx9w\" (UID: \"c7e27a94-0a48-466c-abdb-9f6b0758e358\") " pod="openshift-dns/node-resolver-kgx9w" Sep 18 06:22:22.521048 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.520932 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cfng\" (UniqueName: \"kubernetes.io/projected/e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12-kube-api-access-6cfng\") pod \"tuned-w7cmt\" (UID: \"e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12\") " pod="openshift-cluster-node-tuning-operator/tuned-w7cmt" Sep 18 06:22:22.521048 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.520933 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzdct\" (UniqueName: \"kubernetes.io/projected/eb059951-e263-40da-9155-4fd0cf242820-kube-api-access-gzdct\") pod \"ovnkube-node-hjr4h\" (UID: \"eb059951-e263-40da-9155-4fd0cf242820\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:22.521395 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.521360 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62ffb\" (UniqueName: \"kubernetes.io/projected/4d3101c6-973c-48d6-8efe-f4fab4282824-kube-api-access-62ffb\") pod \"aws-ebs-csi-driver-node-wmmq2\" (UID: \"4d3101c6-973c-48d6-8efe-f4fab4282824\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmmq2" Sep 18 06:22:22.521706 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.521682 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m7w4\" (UniqueName: \"kubernetes.io/projected/0b83c068-4913-4ee5-85ed-17f44dc9f0ca-kube-api-access-8m7w4\") pod \"multus-additional-cni-plugins-q4gmz\" (UID: \"0b83c068-4913-4ee5-85ed-17f44dc9f0ca\") " pod="openshift-multus/multus-additional-cni-plugins-q4gmz" Sep 18 06:22:22.601886 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.601859 2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 18 06:22:22.602936 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.602907 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5dt8f" Sep 18 06:22:22.611958 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.611938 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xccgg" Sep 18 06:22:22.623520 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.623498 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-p94t6" Sep 18 06:22:22.629523 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.629507 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:22.637500 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.637480 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sxrxx" Sep 18 06:22:22.644109 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.644094 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-w7cmt" Sep 18 06:22:22.652023 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.652006 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kgx9w" Sep 18 06:22:22.659735 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.659717 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmmq2" Sep 18 06:22:22.665251 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.665235 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-q4gmz" Sep 18 06:22:22.916145 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:22.916075 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwmtn\" (UniqueName: \"kubernetes.io/projected/ce5f0b70-0633-4019-a805-85d798616ac2-kube-api-access-pwmtn\") pod \"network-check-target-5rhpx\" (UID: \"ce5f0b70-0633-4019-a805-85d798616ac2\") " pod="openshift-network-diagnostics/network-check-target-5rhpx" Sep 18 06:22:22.916271 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:22.916229 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 18 06:22:22.916271 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:22.916249 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 18 06:22:22.916271 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:22.916262 2567 projected.go:196] Error preparing data for projected volume kube-api-access-pwmtn for pod openshift-network-diagnostics/network-check-target-5rhpx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 06:22:22.916398 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:22.916320 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce5f0b70-0633-4019-a805-85d798616ac2-kube-api-access-pwmtn podName:ce5f0b70-0633-4019-a805-85d798616ac2 nodeName:}" failed. No retries permitted until 2026-09-18 06:22:23.916304375 +0000 UTC m=+5.655720540 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-pwmtn" (UniqueName: "kubernetes.io/projected/ce5f0b70-0633-4019-a805-85d798616ac2-kube-api-access-pwmtn") pod "network-check-target-5rhpx" (UID: "ce5f0b70-0633-4019-a805-85d798616ac2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 06:22:23.016613 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:23.016578 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e57e7a14-2ca6-4f41-875f-822739fa1c43-original-pull-secret\") pod \"global-pull-secret-syncer-2jzcv\" (UID: \"e57e7a14-2ca6-4f41-875f-822739fa1c43\") " pod="kube-system/global-pull-secret-syncer-2jzcv" Sep 18 06:22:23.016794 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:23.016641 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66dca255-1289-49ec-b200-e950a30e3d9c-metrics-certs\") pod \"network-metrics-daemon-k77fk\" (UID: \"66dca255-1289-49ec-b200-e950a30e3d9c\") " pod="openshift-multus/network-metrics-daemon-k77fk" Sep 18 06:22:23.016794 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:23.016727 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 06:22:23.016794 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:23.016734 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 18 06:22:23.016794 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:23.016791 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66dca255-1289-49ec-b200-e950a30e3d9c-metrics-certs podName:66dca255-1289-49ec-b200-e950a30e3d9c nodeName:}" failed. No retries permitted until 2026-09-18 06:22:24.016773675 +0000 UTC m=+5.756189830 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66dca255-1289-49ec-b200-e950a30e3d9c-metrics-certs") pod "network-metrics-daemon-k77fk" (UID: "66dca255-1289-49ec-b200-e950a30e3d9c") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 06:22:23.016948 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:23.016807 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e57e7a14-2ca6-4f41-875f-822739fa1c43-original-pull-secret podName:e57e7a14-2ca6-4f41-875f-822739fa1c43 nodeName:}" failed. No retries permitted until 2026-09-18 06:22:24.016799956 +0000 UTC m=+5.756216108 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e57e7a14-2ca6-4f41-875f-822739fa1c43-original-pull-secret") pod "global-pull-secret-syncer-2jzcv" (UID: "e57e7a14-2ca6-4f41-875f-822739fa1c43") : object "kube-system"/"original-pull-secret" not registered Sep 18 06:22:23.244845 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:23.244808 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c2fc9d9_19db_4703_aa25_95b5f5a42e99.slice/crio-f61e6c801b75f72553ca5a071aea94662f767c4f9f927ea2fa18b0cfa902f2e5 WatchSource:0}: Error finding container f61e6c801b75f72553ca5a071aea94662f767c4f9f927ea2fa18b0cfa902f2e5: Status 404 returned error can't find the container with id f61e6c801b75f72553ca5a071aea94662f767c4f9f927ea2fa18b0cfa902f2e5 Sep 18 06:22:23.246357 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:23.246329 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode22cb119_2ea0_4f9e_8067_4a7cd3e2fd12.slice/crio-e39ee8a716acd04c6d759102e56c60e432407c78b9c0b21f160eca6031363e28 WatchSource:0}: Error finding container e39ee8a716acd04c6d759102e56c60e432407c78b9c0b21f160eca6031363e28: Status 404 returned error can't find the container with id e39ee8a716acd04c6d759102e56c60e432407c78b9c0b21f160eca6031363e28 Sep 18 06:22:23.251350 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:23.251324 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b83c068_4913_4ee5_85ed_17f44dc9f0ca.slice/crio-1fdbb8afa956b4971d9a1b916c46bed69504590224c23d08316df99de5e04192 WatchSource:0}: Error finding container 1fdbb8afa956b4971d9a1b916c46bed69504590224c23d08316df99de5e04192: Status 404 returned error can't find the container with id 1fdbb8afa956b4971d9a1b916c46bed69504590224c23d08316df99de5e04192 Sep 18 06:22:23.252432 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:23.252415 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54e09508_bf0d_4f03_b35e_69b4b61d3a9f.slice/crio-88597341c131e455dc1a433560dc6bc73a93917c7c7979fe1a51666bed6c2602 WatchSource:0}: Error finding container 88597341c131e455dc1a433560dc6bc73a93917c7c7979fe1a51666bed6c2602: Status 404 returned error can't find the container with id 88597341c131e455dc1a433560dc6bc73a93917c7c7979fe1a51666bed6c2602 Sep 18 06:22:23.253756 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:23.253727 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7e27a94_0a48_466c_abdb_9f6b0758e358.slice/crio-741608e401d8ea93c6089f01710d7317054bf5768ca082b01813a0d526740766 WatchSource:0}: Error finding container 741608e401d8ea93c6089f01710d7317054bf5768ca082b01813a0d526740766: Status 404 returned error can't find the container with id 741608e401d8ea93c6089f01710d7317054bf5768ca082b01813a0d526740766 Sep 18 06:22:23.254639 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:23.254612 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod631676f2_32af_4b0d_834b_41c7c59db5ba.slice/crio-6659479e327f7a718478e379f62dcbd3559759088cf095f664ad4fa0c252e1b6 WatchSource:0}: Error finding container 6659479e327f7a718478e379f62dcbd3559759088cf095f664ad4fa0c252e1b6: Status 404 returned error can't find the container with id 6659479e327f7a718478e379f62dcbd3559759088cf095f664ad4fa0c252e1b6 Sep 18 06:22:23.256511 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:23.256411 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d3101c6_973c_48d6_8efe_f4fab4282824.slice/crio-073a25d8b3f21d6b86b621083a8a32863622aef322e86bae13f65e6f125fff12 WatchSource:0}: Error finding container 073a25d8b3f21d6b86b621083a8a32863622aef322e86bae13f65e6f125fff12: Status 404 returned error can't find the container with id 073a25d8b3f21d6b86b621083a8a32863622aef322e86bae13f65e6f125fff12 Sep 18 06:22:23.256927 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:23.256907 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb059951_e263_40da_9155_4fd0cf242820.slice/crio-797fa445e75fe6a58ea53694e9a84eb94dca0b25646661cdc425563e9a7ca967 WatchSource:0}: Error finding container 797fa445e75fe6a58ea53694e9a84eb94dca0b25646661cdc425563e9a7ca967: Status 404 returned error can't find the container with id 797fa445e75fe6a58ea53694e9a84eb94dca0b25646661cdc425563e9a7ca967 Sep 18 06:22:23.258455 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:23.258284 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27cccfd2_ff8e_4f71_bc24_5072f09b7f21.slice/crio-554fc1a936771af858e005a761ea00fe8440d2ef1289a1357e6ccb7c0c7d2657 WatchSource:0}: Error finding container 554fc1a936771af858e005a761ea00fe8440d2ef1289a1357e6ccb7c0c7d2657: Status 404 returned error can't find the container with id 554fc1a936771af858e005a761ea00fe8440d2ef1289a1357e6ccb7c0c7d2657 Sep 18 06:22:23.335901 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:23.335869 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 06:17:22 +0000 UTC" deadline="2028-02-11 07:21:38.504966928 +0000 UTC" Sep 18 06:22:23.335901 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:23.335899 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12264h59m15.169071093s" Sep 18 06:22:23.439465 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:23.439439 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2jzcv" Sep 18 06:22:23.439630 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:23.439538 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2jzcv" podUID="e57e7a14-2ca6-4f41-875f-822739fa1c43" Sep 18 06:22:23.452448 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:23.452416 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-104.ec2.internal" event={"ID":"f23d030455f844c3f1c0ef497274c121","Type":"ContainerStarted","Data":"6242969e4c0af60c155f1fe11f0a2e2f51acba33e039e0643baf16916e837288"} Sep 18 06:22:23.453571 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:23.453547 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xccgg" event={"ID":"27cccfd2-ff8e-4f71-bc24-5072f09b7f21","Type":"ContainerStarted","Data":"554fc1a936771af858e005a761ea00fe8440d2ef1289a1357e6ccb7c0c7d2657"} Sep 18 06:22:23.454540 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:23.454522 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" event={"ID":"eb059951-e263-40da-9155-4fd0cf242820","Type":"ContainerStarted","Data":"797fa445e75fe6a58ea53694e9a84eb94dca0b25646661cdc425563e9a7ca967"} Sep 18 06:22:23.455557 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:23.455536 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmmq2" event={"ID":"4d3101c6-973c-48d6-8efe-f4fab4282824","Type":"ContainerStarted","Data":"073a25d8b3f21d6b86b621083a8a32863622aef322e86bae13f65e6f125fff12"} Sep 18 06:22:23.456598 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:23.456577 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kgx9w" event={"ID":"c7e27a94-0a48-466c-abdb-9f6b0758e358","Type":"ContainerStarted","Data":"741608e401d8ea93c6089f01710d7317054bf5768ca082b01813a0d526740766"} Sep 18 06:22:23.457678 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:23.457617 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5dt8f" event={"ID":"54e09508-bf0d-4f03-b35e-69b4b61d3a9f","Type":"ContainerStarted","Data":"88597341c131e455dc1a433560dc6bc73a93917c7c7979fe1a51666bed6c2602"} Sep 18 06:22:23.462440 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:23.462415 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sxrxx" event={"ID":"631676f2-32af-4b0d-834b-41c7c59db5ba","Type":"ContainerStarted","Data":"6659479e327f7a718478e379f62dcbd3559759088cf095f664ad4fa0c252e1b6"} Sep 18 06:22:23.463488 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:23.463465 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q4gmz" event={"ID":"0b83c068-4913-4ee5-85ed-17f44dc9f0ca","Type":"ContainerStarted","Data":"1fdbb8afa956b4971d9a1b916c46bed69504590224c23d08316df99de5e04192"} Sep 18 06:22:23.464494 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:23.464469 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-w7cmt" event={"ID":"e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12","Type":"ContainerStarted","Data":"e39ee8a716acd04c6d759102e56c60e432407c78b9c0b21f160eca6031363e28"} Sep 18 06:22:23.465434 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:23.465347 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-104.ec2.internal" podStartSLOduration=2.465332809 podStartE2EDuration="2.465332809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 06:22:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:22:23.465209564 +0000 UTC m=+5.204625740" watchObservedRunningTime="2026-09-18 06:22:23.465332809 +0000 UTC m=+5.204748983" Sep 18 06:22:23.465585 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:23.465562 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-p94t6" event={"ID":"9c2fc9d9-19db-4703-aa25-95b5f5a42e99","Type":"ContainerStarted","Data":"f61e6c801b75f72553ca5a071aea94662f767c4f9f927ea2fa18b0cfa902f2e5"} Sep 18 06:22:23.837501 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:23.837045 2567 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 18 06:22:23.925284 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:23.924640 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwmtn\" (UniqueName: \"kubernetes.io/projected/ce5f0b70-0633-4019-a805-85d798616ac2-kube-api-access-pwmtn\") pod \"network-check-target-5rhpx\" (UID: \"ce5f0b70-0633-4019-a805-85d798616ac2\") " pod="openshift-network-diagnostics/network-check-target-5rhpx" Sep 18 06:22:23.925284 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:23.924815 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 18 06:22:23.925284 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:23.924834 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 18 06:22:23.925284 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:23.924846 2567 projected.go:196] Error preparing data for projected volume kube-api-access-pwmtn for pod openshift-network-diagnostics/network-check-target-5rhpx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 06:22:23.925284 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:23.924902 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce5f0b70-0633-4019-a805-85d798616ac2-kube-api-access-pwmtn podName:ce5f0b70-0633-4019-a805-85d798616ac2 nodeName:}" failed. No retries permitted until 2026-09-18 06:22:25.924884751 +0000 UTC m=+7.664300920 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-pwmtn" (UniqueName: "kubernetes.io/projected/ce5f0b70-0633-4019-a805-85d798616ac2-kube-api-access-pwmtn") pod "network-check-target-5rhpx" (UID: "ce5f0b70-0633-4019-a805-85d798616ac2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 06:22:24.026349 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:24.025636 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e57e7a14-2ca6-4f41-875f-822739fa1c43-original-pull-secret\") pod \"global-pull-secret-syncer-2jzcv\" (UID: \"e57e7a14-2ca6-4f41-875f-822739fa1c43\") " pod="kube-system/global-pull-secret-syncer-2jzcv" Sep 18 06:22:24.026349 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:24.025726 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66dca255-1289-49ec-b200-e950a30e3d9c-metrics-certs\") pod \"network-metrics-daemon-k77fk\" (UID: \"66dca255-1289-49ec-b200-e950a30e3d9c\") " pod="openshift-multus/network-metrics-daemon-k77fk" Sep 18 06:22:24.026349 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:24.025857 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 06:22:24.026349 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:24.025918 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66dca255-1289-49ec-b200-e950a30e3d9c-metrics-certs podName:66dca255-1289-49ec-b200-e950a30e3d9c nodeName:}" failed. No retries permitted until 2026-09-18 06:22:26.025899947 +0000 UTC m=+7.765316112 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66dca255-1289-49ec-b200-e950a30e3d9c-metrics-certs") pod "network-metrics-daemon-k77fk" (UID: "66dca255-1289-49ec-b200-e950a30e3d9c") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 06:22:24.026349 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:24.025982 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 18 06:22:24.026349 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:24.026014 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e57e7a14-2ca6-4f41-875f-822739fa1c43-original-pull-secret podName:e57e7a14-2ca6-4f41-875f-822739fa1c43 nodeName:}" failed. No retries permitted until 2026-09-18 06:22:26.026002836 +0000 UTC m=+7.765418993 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e57e7a14-2ca6-4f41-875f-822739fa1c43-original-pull-secret") pod "global-pull-secret-syncer-2jzcv" (UID: "e57e7a14-2ca6-4f41-875f-822739fa1c43") : object "kube-system"/"original-pull-secret" not registered Sep 18 06:22:24.336487 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:24.336335 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 06:17:22 +0000 UTC" deadline="2028-06-30 19:15:12.284762 +0000 UTC" Sep 18 06:22:24.336487 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:24.336440 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15636h52m47.948327676s" Sep 18 06:22:24.441044 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:24.440479 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k77fk" Sep 18 06:22:24.441044 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:24.440619 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k77fk" podUID="66dca255-1289-49ec-b200-e950a30e3d9c" Sep 18 06:22:24.441545 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:24.441345 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5rhpx" Sep 18 06:22:24.441545 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:24.441471 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5rhpx" podUID="ce5f0b70-0633-4019-a805-85d798616ac2" Sep 18 06:22:24.481206 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:24.480753 2567 generic.go:356] "Generic (PLEG): container finished" podID="093f6c5a5a5dfa9d45afb8587676bdd0" containerID="72af7ef3f6060eee2d4513c7098b363092625437690ecac0b83737f272fcdea3" exitCode=0 Sep 18 06:22:24.481206 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:24.480902 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-104.ec2.internal" event={"ID":"093f6c5a5a5dfa9d45afb8587676bdd0","Type":"ContainerDied","Data":"72af7ef3f6060eee2d4513c7098b363092625437690ecac0b83737f272fcdea3"} Sep 18 06:22:25.440048 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:25.439565 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2jzcv" Sep 18 06:22:25.440048 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:25.439707 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2jzcv" podUID="e57e7a14-2ca6-4f41-875f-822739fa1c43" Sep 18 06:22:25.503394 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:25.502954 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-104.ec2.internal" event={"ID":"093f6c5a5a5dfa9d45afb8587676bdd0","Type":"ContainerStarted","Data":"15d50904d70166c69f98fab3015e77479633977f9b18033a3f0a83d20b8036f3"} Sep 18 06:22:25.519267 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:25.518943 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-104.ec2.internal" podStartSLOduration=4.518928955 podStartE2EDuration="4.518928955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 06:22:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:22:25.518409682 +0000 UTC m=+7.257825868" watchObservedRunningTime="2026-09-18 06:22:25.518928955 +0000 UTC m=+7.258345129" Sep 18 06:22:25.941627 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:25.940904 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwmtn\" (UniqueName: \"kubernetes.io/projected/ce5f0b70-0633-4019-a805-85d798616ac2-kube-api-access-pwmtn\") pod \"network-check-target-5rhpx\" (UID: \"ce5f0b70-0633-4019-a805-85d798616ac2\") " pod="openshift-network-diagnostics/network-check-target-5rhpx" Sep 18 06:22:25.941627 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:25.941094 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 18 06:22:25.941627 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:25.941114 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 18 06:22:25.941627 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:25.941127 2567 projected.go:196] Error preparing data for projected volume kube-api-access-pwmtn for pod openshift-network-diagnostics/network-check-target-5rhpx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 06:22:25.941627 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:25.941185 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce5f0b70-0633-4019-a805-85d798616ac2-kube-api-access-pwmtn podName:ce5f0b70-0633-4019-a805-85d798616ac2 nodeName:}" failed. No retries permitted until 2026-09-18 06:22:29.94116697 +0000 UTC m=+11.680583129 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-pwmtn" (UniqueName: "kubernetes.io/projected/ce5f0b70-0633-4019-a805-85d798616ac2-kube-api-access-pwmtn") pod "network-check-target-5rhpx" (UID: "ce5f0b70-0633-4019-a805-85d798616ac2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 06:22:26.041807 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:26.041766 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66dca255-1289-49ec-b200-e950a30e3d9c-metrics-certs\") pod \"network-metrics-daemon-k77fk\" (UID: \"66dca255-1289-49ec-b200-e950a30e3d9c\") " pod="openshift-multus/network-metrics-daemon-k77fk" Sep 18 06:22:26.042009 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:26.041845 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e57e7a14-2ca6-4f41-875f-822739fa1c43-original-pull-secret\") pod \"global-pull-secret-syncer-2jzcv\" (UID: \"e57e7a14-2ca6-4f41-875f-822739fa1c43\") " pod="kube-system/global-pull-secret-syncer-2jzcv" Sep 18 06:22:26.042093 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:26.042015 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 18 06:22:26.042093 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:26.042079 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e57e7a14-2ca6-4f41-875f-822739fa1c43-original-pull-secret podName:e57e7a14-2ca6-4f41-875f-822739fa1c43 nodeName:}" failed. No retries permitted until 2026-09-18 06:22:30.042060855 +0000 UTC m=+11.781477012 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e57e7a14-2ca6-4f41-875f-822739fa1c43-original-pull-secret") pod "global-pull-secret-syncer-2jzcv" (UID: "e57e7a14-2ca6-4f41-875f-822739fa1c43") : object "kube-system"/"original-pull-secret" not registered Sep 18 06:22:26.042546 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:26.042504 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 06:22:26.042631 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:26.042563 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66dca255-1289-49ec-b200-e950a30e3d9c-metrics-certs podName:66dca255-1289-49ec-b200-e950a30e3d9c nodeName:}" failed. No retries permitted until 2026-09-18 06:22:30.042547606 +0000 UTC m=+11.781963761 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66dca255-1289-49ec-b200-e950a30e3d9c-metrics-certs") pod "network-metrics-daemon-k77fk" (UID: "66dca255-1289-49ec-b200-e950a30e3d9c") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 06:22:26.443279 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:26.440577 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5rhpx" Sep 18 06:22:26.443279 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:26.440701 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5rhpx" podUID="ce5f0b70-0633-4019-a805-85d798616ac2" Sep 18 06:22:26.443279 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:26.440538 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k77fk" Sep 18 06:22:26.443279 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:26.443183 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k77fk" podUID="66dca255-1289-49ec-b200-e950a30e3d9c" Sep 18 06:22:27.439516 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:27.439483 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2jzcv" Sep 18 06:22:27.439686 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:27.439610 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2jzcv" podUID="e57e7a14-2ca6-4f41-875f-822739fa1c43" Sep 18 06:22:28.442835 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:28.442803 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5rhpx" Sep 18 06:22:28.443288 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:28.442921 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5rhpx" podUID="ce5f0b70-0633-4019-a805-85d798616ac2" Sep 18 06:22:28.443288 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:28.443000 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k77fk" Sep 18 06:22:28.443288 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:28.443085 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k77fk" podUID="66dca255-1289-49ec-b200-e950a30e3d9c" Sep 18 06:22:29.440001 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:29.439968 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2jzcv" Sep 18 06:22:29.440183 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:29.440099 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2jzcv" podUID="e57e7a14-2ca6-4f41-875f-822739fa1c43" Sep 18 06:22:29.977822 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:29.977780 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwmtn\" (UniqueName: \"kubernetes.io/projected/ce5f0b70-0633-4019-a805-85d798616ac2-kube-api-access-pwmtn\") pod \"network-check-target-5rhpx\" (UID: \"ce5f0b70-0633-4019-a805-85d798616ac2\") " pod="openshift-network-diagnostics/network-check-target-5rhpx" Sep 18 06:22:29.978291 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:29.977972 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 18 06:22:29.978291 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:29.977999 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 18 06:22:29.978291 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:29.978013 2567 projected.go:196] Error preparing data for projected volume kube-api-access-pwmtn for pod openshift-network-diagnostics/network-check-target-5rhpx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 06:22:29.978291 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:29.978084 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce5f0b70-0633-4019-a805-85d798616ac2-kube-api-access-pwmtn podName:ce5f0b70-0633-4019-a805-85d798616ac2 nodeName:}" failed. No retries permitted until 2026-09-18 06:22:37.978064144 +0000 UTC m=+19.717480298 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-pwmtn" (UniqueName: "kubernetes.io/projected/ce5f0b70-0633-4019-a805-85d798616ac2-kube-api-access-pwmtn") pod "network-check-target-5rhpx" (UID: "ce5f0b70-0633-4019-a805-85d798616ac2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 06:22:30.078466 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:30.078419 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66dca255-1289-49ec-b200-e950a30e3d9c-metrics-certs\") pod \"network-metrics-daemon-k77fk\" (UID: \"66dca255-1289-49ec-b200-e950a30e3d9c\") " pod="openshift-multus/network-metrics-daemon-k77fk" Sep 18 06:22:30.078623 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:30.078487 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e57e7a14-2ca6-4f41-875f-822739fa1c43-original-pull-secret\") pod \"global-pull-secret-syncer-2jzcv\" (UID: \"e57e7a14-2ca6-4f41-875f-822739fa1c43\") " pod="kube-system/global-pull-secret-syncer-2jzcv" Sep 18 06:22:30.078623 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:30.078609 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 06:22:30.078707 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:30.078685 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66dca255-1289-49ec-b200-e950a30e3d9c-metrics-certs podName:66dca255-1289-49ec-b200-e950a30e3d9c nodeName:}" failed. No retries permitted until 2026-09-18 06:22:38.078665214 +0000 UTC m=+19.818081388 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66dca255-1289-49ec-b200-e950a30e3d9c-metrics-certs") pod "network-metrics-daemon-k77fk" (UID: "66dca255-1289-49ec-b200-e950a30e3d9c") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 06:22:30.079005 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:30.078620 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 18 06:22:30.079081 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:30.079049 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e57e7a14-2ca6-4f41-875f-822739fa1c43-original-pull-secret podName:e57e7a14-2ca6-4f41-875f-822739fa1c43 nodeName:}" failed. No retries permitted until 2026-09-18 06:22:38.079030478 +0000 UTC m=+19.818446651 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e57e7a14-2ca6-4f41-875f-822739fa1c43-original-pull-secret") pod "global-pull-secret-syncer-2jzcv" (UID: "e57e7a14-2ca6-4f41-875f-822739fa1c43") : object "kube-system"/"original-pull-secret" not registered Sep 18 06:22:30.443462 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:30.442948 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5rhpx" Sep 18 06:22:30.443462 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:30.442967 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k77fk" Sep 18 06:22:30.443462 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:30.443065 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5rhpx" podUID="ce5f0b70-0633-4019-a805-85d798616ac2" Sep 18 06:22:30.443462 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:30.443187 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k77fk" podUID="66dca255-1289-49ec-b200-e950a30e3d9c" Sep 18 06:22:31.439336 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:31.439247 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2jzcv" Sep 18 06:22:31.439933 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:31.439399 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2jzcv" podUID="e57e7a14-2ca6-4f41-875f-822739fa1c43" Sep 18 06:22:32.443430 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:32.443397 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5rhpx" Sep 18 06:22:32.443794 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:32.443519 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5rhpx" podUID="ce5f0b70-0633-4019-a805-85d798616ac2" Sep 18 06:22:32.443794 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:32.443578 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k77fk" Sep 18 06:22:32.443794 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:32.443678 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k77fk" podUID="66dca255-1289-49ec-b200-e950a30e3d9c" Sep 18 06:22:33.439748 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:33.439665 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2jzcv" Sep 18 06:22:33.439908 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:33.439795 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2jzcv" podUID="e57e7a14-2ca6-4f41-875f-822739fa1c43" Sep 18 06:22:34.443235 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:34.443203 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k77fk" Sep 18 06:22:34.443235 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:34.443239 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5rhpx" Sep 18 06:22:34.443676 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:34.443326 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k77fk" podUID="66dca255-1289-49ec-b200-e950a30e3d9c" Sep 18 06:22:34.443676 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:34.443431 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5rhpx" podUID="ce5f0b70-0633-4019-a805-85d798616ac2" Sep 18 06:22:35.440106 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:35.440075 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2jzcv" Sep 18 06:22:35.440250 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:35.440226 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2jzcv" podUID="e57e7a14-2ca6-4f41-875f-822739fa1c43" Sep 18 06:22:36.440133 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:36.440096 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k77fk" Sep 18 06:22:36.440596 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:36.440216 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k77fk" podUID="66dca255-1289-49ec-b200-e950a30e3d9c" Sep 18 06:22:36.440596 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:36.440275 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5rhpx" Sep 18 06:22:36.440596 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:36.440406 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5rhpx" podUID="ce5f0b70-0633-4019-a805-85d798616ac2" Sep 18 06:22:37.439633 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:37.439594 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2jzcv" Sep 18 06:22:37.439808 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:37.439709 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2jzcv" podUID="e57e7a14-2ca6-4f41-875f-822739fa1c43" Sep 18 06:22:38.036682 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:38.036641 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwmtn\" (UniqueName: \"kubernetes.io/projected/ce5f0b70-0633-4019-a805-85d798616ac2-kube-api-access-pwmtn\") pod \"network-check-target-5rhpx\" (UID: \"ce5f0b70-0633-4019-a805-85d798616ac2\") " pod="openshift-network-diagnostics/network-check-target-5rhpx" Sep 18 06:22:38.037108 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:38.036826 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 18 06:22:38.037108 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:38.036857 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 18 06:22:38.037108 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:38.036872 2567 projected.go:196] Error preparing data for projected volume kube-api-access-pwmtn for pod openshift-network-diagnostics/network-check-target-5rhpx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 06:22:38.037108 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:38.036943 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce5f0b70-0633-4019-a805-85d798616ac2-kube-api-access-pwmtn podName:ce5f0b70-0633-4019-a805-85d798616ac2 nodeName:}" failed. No retries permitted until 2026-09-18 06:22:54.036923106 +0000 UTC m=+35.776339278 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-pwmtn" (UniqueName: "kubernetes.io/projected/ce5f0b70-0633-4019-a805-85d798616ac2-kube-api-access-pwmtn") pod "network-check-target-5rhpx" (UID: "ce5f0b70-0633-4019-a805-85d798616ac2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 06:22:38.137347 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:38.137313 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e57e7a14-2ca6-4f41-875f-822739fa1c43-original-pull-secret\") pod \"global-pull-secret-syncer-2jzcv\" (UID: \"e57e7a14-2ca6-4f41-875f-822739fa1c43\") " pod="kube-system/global-pull-secret-syncer-2jzcv" Sep 18 06:22:38.137528 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:38.137408 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66dca255-1289-49ec-b200-e950a30e3d9c-metrics-certs\") pod \"network-metrics-daemon-k77fk\" (UID: \"66dca255-1289-49ec-b200-e950a30e3d9c\") " pod="openshift-multus/network-metrics-daemon-k77fk" Sep 18 06:22:38.137528 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:38.137478 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 18 06:22:38.137528 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:38.137493 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 06:22:38.137671 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:38.137542 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e57e7a14-2ca6-4f41-875f-822739fa1c43-original-pull-secret podName:e57e7a14-2ca6-4f41-875f-822739fa1c43 nodeName:}" failed. No retries permitted until 2026-09-18 06:22:54.137524022 +0000 UTC m=+35.876940188 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e57e7a14-2ca6-4f41-875f-822739fa1c43-original-pull-secret") pod "global-pull-secret-syncer-2jzcv" (UID: "e57e7a14-2ca6-4f41-875f-822739fa1c43") : object "kube-system"/"original-pull-secret" not registered Sep 18 06:22:38.137671 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:38.137559 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66dca255-1289-49ec-b200-e950a30e3d9c-metrics-certs podName:66dca255-1289-49ec-b200-e950a30e3d9c nodeName:}" failed. No retries permitted until 2026-09-18 06:22:54.137550718 +0000 UTC m=+35.876966870 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66dca255-1289-49ec-b200-e950a30e3d9c-metrics-certs") pod "network-metrics-daemon-k77fk" (UID: "66dca255-1289-49ec-b200-e950a30e3d9c") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 06:22:38.439625 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:38.439550 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5rhpx" Sep 18 06:22:38.439784 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:38.439552 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k77fk" Sep 18 06:22:38.439784 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:38.439675 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5rhpx" podUID="ce5f0b70-0633-4019-a805-85d798616ac2" Sep 18 06:22:38.439784 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:38.439756 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k77fk" podUID="66dca255-1289-49ec-b200-e950a30e3d9c" Sep 18 06:22:39.440020 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:39.439990 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2jzcv" Sep 18 06:22:39.440453 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:39.440116 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2jzcv" podUID="e57e7a14-2ca6-4f41-875f-822739fa1c43" Sep 18 06:22:40.440261 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:40.440229 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k77fk" Sep 18 06:22:40.440669 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:40.440341 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k77fk" podUID="66dca255-1289-49ec-b200-e950a30e3d9c" Sep 18 06:22:40.440669 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:40.440385 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5rhpx" Sep 18 06:22:40.440669 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:40.440464 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5rhpx" podUID="ce5f0b70-0633-4019-a805-85d798616ac2" Sep 18 06:22:41.440278 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:41.439856 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2jzcv" Sep 18 06:22:41.440997 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:41.440330 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2jzcv" podUID="e57e7a14-2ca6-4f41-875f-822739fa1c43" Sep 18 06:22:41.533256 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:41.533217 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xccgg" event={"ID":"27cccfd2-ff8e-4f71-bc24-5072f09b7f21","Type":"ContainerStarted","Data":"f7cbec284161b2b3afc54e40b15abfa269312ee9a66593f61f280d76e0a18fdb"} Sep 18 06:22:41.536576 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:41.536551 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjr4h_eb059951-e263-40da-9155-4fd0cf242820/ovn-acl-logging/0.log" Sep 18 06:22:41.536954 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:41.536928 2567 generic.go:356] "Generic (PLEG): container finished" podID="eb059951-e263-40da-9155-4fd0cf242820" containerID="1f002eca093301a488e46b18391b91ed6adff3540c9a00a68c358e8390c429c7" exitCode=1 Sep 18 06:22:41.537082 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:41.536995 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" event={"ID":"eb059951-e263-40da-9155-4fd0cf242820","Type":"ContainerStarted","Data":"3164fbe24b66ef59ed93d1f16270441a4e71563bf94152433bef62ee9253c744"} Sep 18 06:22:41.537082 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:41.537074 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" event={"ID":"eb059951-e263-40da-9155-4fd0cf242820","Type":"ContainerStarted","Data":"b96e95ed1cb839b328d1cd1a695e52049ec307c92e0877e5f83d35d116634413"} Sep 18 06:22:41.537186 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:41.537114 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" event={"ID":"eb059951-e263-40da-9155-4fd0cf242820","Type":"ContainerStarted","Data":"ad22c15508f0530d4d693df79c20b7a6ac5fe21022c13daa673447e2e24c01d1"} Sep 18 06:22:41.537186 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:41.537143 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" event={"ID":"eb059951-e263-40da-9155-4fd0cf242820","Type":"ContainerStarted","Data":"d6284ae35e940c8272bf78574b60e0e65ab3e85380f0a1c66bb8f42945768bc0"} Sep 18 06:22:41.537186 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:41.537178 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" event={"ID":"eb059951-e263-40da-9155-4fd0cf242820","Type":"ContainerDied","Data":"1f002eca093301a488e46b18391b91ed6adff3540c9a00a68c358e8390c429c7"} Sep 18 06:22:41.537318 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:41.537225 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" event={"ID":"eb059951-e263-40da-9155-4fd0cf242820","Type":"ContainerStarted","Data":"39cd8e12572efe37536a4dadc7864149512aca1f2f7dee740dd43e2993631e8f"} Sep 18 06:22:41.538737 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:41.538709 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmmq2" event={"ID":"4d3101c6-973c-48d6-8efe-f4fab4282824","Type":"ContainerStarted","Data":"245c1f75ffd6459e5a1602f447dfed42ba02e973586c5881338f1c71a582dcf1"} Sep 18 06:22:41.539979 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:41.539955 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kgx9w" event={"ID":"c7e27a94-0a48-466c-abdb-9f6b0758e358","Type":"ContainerStarted","Data":"3e681e000e954717823d207bea6e26b5b5c6d54da12056d346385bb4f2947f82"} Sep 18 06:22:41.541414 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:41.541389 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5dt8f" event={"ID":"54e09508-bf0d-4f03-b35e-69b4b61d3a9f","Type":"ContainerStarted","Data":"12187f7a2b7368626fb9cbcdc9a0b7f2494c85421501da08a67f693800d90a95"} Sep 18 06:22:41.542643 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:41.542624 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sxrxx" event={"ID":"631676f2-32af-4b0d-834b-41c7c59db5ba","Type":"ContainerStarted","Data":"80f4bc27feac41d942eae1fc64e1c6d13f4aa767dce6013e8c3c91ee72b98b70"} Sep 18 06:22:41.544061 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:41.544043 2567 generic.go:356] "Generic (PLEG): container finished" podID="0b83c068-4913-4ee5-85ed-17f44dc9f0ca" containerID="16f0b75e39c1975a261a1ac3133d40d6eea620505cf299e9ba3e71b7902efbb6" exitCode=0 Sep 18 06:22:41.544143 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:41.544098 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q4gmz" event={"ID":"0b83c068-4913-4ee5-85ed-17f44dc9f0ca","Type":"ContainerDied","Data":"16f0b75e39c1975a261a1ac3133d40d6eea620505cf299e9ba3e71b7902efbb6"} Sep 18 06:22:41.545521 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:41.545506 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-w7cmt" event={"ID":"e22cb119-2ea0-4f9e-8067-4a7cd3e2fd12","Type":"ContainerStarted","Data":"26c80f0cc0c1fcf37dc54d235a6b5e2a21921ab30dbfe7052dcd216ecb63526e"} Sep 18 06:22:41.546765 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:41.546730 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-xccgg" podStartSLOduration=4.152694219 podStartE2EDuration="21.546706255s" totalImagesPullingTime="17.394012036s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 06:22:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:22:41.546513937 +0000 UTC m=+23.285930129" watchObservedRunningTime="2026-09-18 06:22:41.546706255 +0000 UTC m=+23.286122424" Sep 18 06:22:41.585635 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:41.585592 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-5dt8f" podStartSLOduration=4.107853556 podStartE2EDuration="21.585574795s" totalImagesPullingTime="17.477721239s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 06:22:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:22:41.563196468 +0000 UTC m=+23.302612662" watchObservedRunningTime="2026-09-18 06:22:41.585574795 +0000 UTC m=+23.324990969" Sep 18 06:22:41.600035 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:41.599991 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-sxrxx" podStartSLOduration=4.643729631 podStartE2EDuration="21.599978452s" totalImagesPullingTime="16.956248821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:22:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:22:41.599739346 +0000 UTC m=+23.339155539" watchObservedRunningTime="2026-09-18 06:22:41.599978452 +0000 UTC m=+23.339394626" Sep 18 06:22:41.614769 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:41.614732 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-w7cmt" podStartSLOduration=4.178390607 podStartE2EDuration="21.614721354s" totalImagesPullingTime="17.436330747s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 06:22:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:22:41.614703212 +0000 UTC m=+23.354119405" watchObservedRunningTime="2026-09-18 06:22:41.614721354 +0000 UTC m=+23.354137527" Sep 18 06:22:41.634748 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:41.634707 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-kgx9w" podStartSLOduration=4.236298344 podStartE2EDuration="21.634693994s" totalImagesPullingTime="17.39839565s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 06:22:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:22:41.634114886 +0000 UTC m=+23.373531064" watchObservedRunningTime="2026-09-18 06:22:41.634693994 +0000 UTC m=+23.374110167" Sep 18 06:22:42.062382 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:42.062341 2567 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 18 06:22:42.381171 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:42.381035 2567 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-18T06:22:42.062374758Z","UUID":"3ec8c0c0-267f-4458-ac70-89320b7de721","Handler":null,"Name":"","Endpoint":""} Sep 18 06:22:42.382398 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:42.382385 2567 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 18 06:22:42.382480 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:42.382412 2567 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 18 06:22:42.440287 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:42.440262 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5rhpx" Sep 18 06:22:42.440287 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:42.440295 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k77fk" Sep 18 06:22:42.440745 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:42.440391 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5rhpx" podUID="ce5f0b70-0633-4019-a805-85d798616ac2" Sep 18 06:22:42.440745 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:42.440536 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k77fk" podUID="66dca255-1289-49ec-b200-e950a30e3d9c" Sep 18 06:22:42.550280 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:42.550242 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmmq2" event={"ID":"4d3101c6-973c-48d6-8efe-f4fab4282824","Type":"ContainerStarted","Data":"d8ddbbccffc4e2f319a0381a1c0b5e8f6ded10582d9d5155e91586d099c79d5c"} Sep 18 06:22:42.552605 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:42.552071 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-p94t6" event={"ID":"9c2fc9d9-19db-4703-aa25-95b5f5a42e99","Type":"ContainerStarted","Data":"3d0bc784b9f904e97df38df980bb1587305974dd0c736711c913dd7dd9fe7352"} Sep 18 06:22:43.439265 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:43.439240 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2jzcv" Sep 18 06:22:43.439413 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:43.439332 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2jzcv" podUID="e57e7a14-2ca6-4f41-875f-822739fa1c43" Sep 18 06:22:43.558972 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:43.558950 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjr4h_eb059951-e263-40da-9155-4fd0cf242820/ovn-acl-logging/0.log" Sep 18 06:22:43.559706 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:43.559674 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" event={"ID":"eb059951-e263-40da-9155-4fd0cf242820","Type":"ContainerStarted","Data":"9b998b60799a3b101953089fb55b8ae0ec11881743657c8d9353897f0ab0e5db"} Sep 18 06:22:43.562392 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:43.562353 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmmq2" event={"ID":"4d3101c6-973c-48d6-8efe-f4fab4282824","Type":"ContainerStarted","Data":"9f3c10a5b1e4bb9e3044f68729e02331f1f101d975b9339000f605d9526268c8"} Sep 18 06:22:43.581849 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:43.581809 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmmq2" podStartSLOduration=4.239947783 podStartE2EDuration="23.581795978s" totalImagesPullingTime="19.341848195s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 06:22:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:22:43.580301465 +0000 UTC m=+25.319717638" watchObservedRunningTime="2026-09-18 06:22:43.581795978 +0000 UTC m=+25.321212151" Sep 18 06:22:43.582002 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:43.581981 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-p94t6" podStartSLOduration=6.141714329 podStartE2EDuration="23.58197601s" totalImagesPullingTime="17.440261681s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 06:22:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:22:42.566092424 +0000 UTC m=+24.305508597" watchObservedRunningTime="2026-09-18 06:22:43.58197601 +0000 UTC m=+25.321392183" Sep 18 06:22:44.440249 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:44.440212 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k77fk" Sep 18 06:22:44.440249 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:44.440249 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5rhpx" Sep 18 06:22:44.440509 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:44.440442 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k77fk" podUID="66dca255-1289-49ec-b200-e950a30e3d9c" Sep 18 06:22:44.440573 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:44.440504 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5rhpx" podUID="ce5f0b70-0633-4019-a805-85d798616ac2" Sep 18 06:22:45.440172 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:45.440140 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2jzcv" Sep 18 06:22:45.440601 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:45.440268 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2jzcv" podUID="e57e7a14-2ca6-4f41-875f-822739fa1c43" Sep 18 06:22:45.573704 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:45.573672 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-sxrxx" Sep 18 06:22:45.574251 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:45.574230 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-sxrxx" Sep 18 06:22:46.439529 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:46.439302 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k77fk" Sep 18 06:22:46.439633 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:46.439352 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5rhpx" Sep 18 06:22:46.439633 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:46.439590 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k77fk" podUID="66dca255-1289-49ec-b200-e950a30e3d9c" Sep 18 06:22:46.439703 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:46.439633 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5rhpx" podUID="ce5f0b70-0633-4019-a805-85d798616ac2" Sep 18 06:22:46.572281 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:46.572253 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjr4h_eb059951-e263-40da-9155-4fd0cf242820/ovn-acl-logging/0.log" Sep 18 06:22:46.573039 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:46.572638 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" event={"ID":"eb059951-e263-40da-9155-4fd0cf242820","Type":"ContainerStarted","Data":"d5230dc4dd1df68450fc98e47f868923377ce3a223e8fea72de1c91b2e0c0bde"} Sep 18 06:22:46.573039 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:46.572806 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:46.573039 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:46.572914 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:46.573039 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:46.573031 2567 scope.go:117] "RemoveContainer" containerID="1f002eca093301a488e46b18391b91ed6adff3540c9a00a68c358e8390c429c7" Sep 18 06:22:46.574651 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:46.574626 2567 generic.go:356] "Generic (PLEG): container finished" podID="0b83c068-4913-4ee5-85ed-17f44dc9f0ca" containerID="bf1d68ebc0db855f2dbdf2851f041bf3a2578c8cd41e0b8243e9e758797490f9" exitCode=0 Sep 18 06:22:46.574743 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:46.574707 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q4gmz" event={"ID":"0b83c068-4913-4ee5-85ed-17f44dc9f0ca","Type":"ContainerDied","Data":"bf1d68ebc0db855f2dbdf2851f041bf3a2578c8cd41e0b8243e9e758797490f9"} Sep 18 06:22:46.574811 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:46.574795 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-sxrxx" Sep 18 06:22:46.575387 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:46.575350 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sxrxx" Sep 18 06:22:46.588576 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:46.588560 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:47.440273 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:47.440108 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2jzcv" Sep 18 06:22:47.440440 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:47.440387 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2jzcv" podUID="e57e7a14-2ca6-4f41-875f-822739fa1c43" Sep 18 06:22:47.582387 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:47.582349 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjr4h_eb059951-e263-40da-9155-4fd0cf242820/ovn-acl-logging/0.log" Sep 18 06:22:47.583156 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:47.583117 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" event={"ID":"eb059951-e263-40da-9155-4fd0cf242820","Type":"ContainerStarted","Data":"bc10d2fc026c5704c01ea4e947822879b889a1a6592aa173d89f311d8d3a261f"} Sep 18 06:22:47.583648 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:47.583622 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:47.599977 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:47.599955 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:22:47.613456 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:47.610490 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" podStartSLOduration=10.173088457 podStartE2EDuration="27.610471954s" totalImagesPullingTime="17.437383497s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 06:22:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:22:47.609087183 +0000 UTC m=+29.348503375" watchObservedRunningTime="2026-09-18 06:22:47.610471954 +0000 UTC m=+29.349888128" Sep 18 06:22:47.770180 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:47.770108 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2jzcv"] Sep 18 06:22:47.770327 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:47.770264 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2jzcv" Sep 18 06:22:47.770417 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:47.770390 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2jzcv" podUID="e57e7a14-2ca6-4f41-875f-822739fa1c43" Sep 18 06:22:47.770746 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:47.770727 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5rhpx"] Sep 18 06:22:47.770825 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:47.770814 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5rhpx" Sep 18 06:22:47.770900 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:47.770882 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5rhpx" podUID="ce5f0b70-0633-4019-a805-85d798616ac2" Sep 18 06:22:47.772604 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:47.772575 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-k77fk"] Sep 18 06:22:47.772697 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:47.772666 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k77fk" Sep 18 06:22:47.772761 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:47.772744 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k77fk" podUID="66dca255-1289-49ec-b200-e950a30e3d9c" Sep 18 06:22:48.587339 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:48.587309 2567 generic.go:356] "Generic (PLEG): container finished" podID="0b83c068-4913-4ee5-85ed-17f44dc9f0ca" containerID="e7072c8983a2f21391252fbfa4087e43eb86ff9cf17b7eb13f068ad8699d423a" exitCode=0 Sep 18 06:22:48.587779 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:48.587393 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q4gmz" event={"ID":"0b83c068-4913-4ee5-85ed-17f44dc9f0ca","Type":"ContainerDied","Data":"e7072c8983a2f21391252fbfa4087e43eb86ff9cf17b7eb13f068ad8699d423a"} Sep 18 06:22:49.440136 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:49.440105 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2jzcv" Sep 18 06:22:49.440136 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:49.440136 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k77fk" Sep 18 06:22:49.440357 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:49.440179 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5rhpx" Sep 18 06:22:49.440357 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:49.440195 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2jzcv" podUID="e57e7a14-2ca6-4f41-875f-822739fa1c43" Sep 18 06:22:49.440357 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:49.440240 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5rhpx" podUID="ce5f0b70-0633-4019-a805-85d798616ac2" Sep 18 06:22:49.440357 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:49.440340 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k77fk" podUID="66dca255-1289-49ec-b200-e950a30e3d9c" Sep 18 06:22:50.594467 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:50.594434 2567 generic.go:356] "Generic (PLEG): container finished" podID="0b83c068-4913-4ee5-85ed-17f44dc9f0ca" containerID="02da5b4a2978e918002054814424356f31208c2cbb1eee6db5aec9b9350265fd" exitCode=0 Sep 18 06:22:50.594799 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:50.594495 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q4gmz" event={"ID":"0b83c068-4913-4ee5-85ed-17f44dc9f0ca","Type":"ContainerDied","Data":"02da5b4a2978e918002054814424356f31208c2cbb1eee6db5aec9b9350265fd"} Sep 18 06:22:51.439811 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:51.439781 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2jzcv" Sep 18 06:22:51.439982 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:51.439782 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5rhpx" Sep 18 06:22:51.439982 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:51.439907 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2jzcv" podUID="e57e7a14-2ca6-4f41-875f-822739fa1c43" Sep 18 06:22:51.439982 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:51.439922 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k77fk" Sep 18 06:22:51.440159 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:51.440015 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5rhpx" podUID="ce5f0b70-0633-4019-a805-85d798616ac2" Sep 18 06:22:51.440159 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:51.440118 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k77fk" podUID="66dca255-1289-49ec-b200-e950a30e3d9c" Sep 18 06:22:53.440147 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:53.439898 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5rhpx" Sep 18 06:22:53.440584 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:53.439925 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k77fk" Sep 18 06:22:53.440584 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:53.440244 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5rhpx" podUID="ce5f0b70-0633-4019-a805-85d798616ac2" Sep 18 06:22:53.440584 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:53.439925 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2jzcv" Sep 18 06:22:53.440584 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:53.440384 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k77fk" podUID="66dca255-1289-49ec-b200-e950a30e3d9c" Sep 18 06:22:53.440584 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:53.440463 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2jzcv" podUID="e57e7a14-2ca6-4f41-875f-822739fa1c43" Sep 18 06:22:54.063632 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:54.063602 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwmtn\" (UniqueName: \"kubernetes.io/projected/ce5f0b70-0633-4019-a805-85d798616ac2-kube-api-access-pwmtn\") pod \"network-check-target-5rhpx\" (UID: \"ce5f0b70-0633-4019-a805-85d798616ac2\") " pod="openshift-network-diagnostics/network-check-target-5rhpx" Sep 18 06:22:54.063843 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:54.063779 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 18 06:22:54.063843 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:54.063803 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 18 06:22:54.063843 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:54.063816 2567 projected.go:196] Error preparing data for projected volume kube-api-access-pwmtn for pod openshift-network-diagnostics/network-check-target-5rhpx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 06:22:54.064010 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:54.063878 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce5f0b70-0633-4019-a805-85d798616ac2-kube-api-access-pwmtn podName:ce5f0b70-0633-4019-a805-85d798616ac2 nodeName:}" failed. No retries permitted until 2026-09-18 06:23:26.063859971 +0000 UTC m=+67.803276132 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-pwmtn" (UniqueName: "kubernetes.io/projected/ce5f0b70-0633-4019-a805-85d798616ac2-kube-api-access-pwmtn") pod "network-check-target-5rhpx" (UID: "ce5f0b70-0633-4019-a805-85d798616ac2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 06:22:54.164713 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:54.164676 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e57e7a14-2ca6-4f41-875f-822739fa1c43-original-pull-secret\") pod \"global-pull-secret-syncer-2jzcv\" (UID: \"e57e7a14-2ca6-4f41-875f-822739fa1c43\") " pod="kube-system/global-pull-secret-syncer-2jzcv" Sep 18 06:22:54.164883 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:54.164767 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66dca255-1289-49ec-b200-e950a30e3d9c-metrics-certs\") pod \"network-metrics-daemon-k77fk\" (UID: \"66dca255-1289-49ec-b200-e950a30e3d9c\") " pod="openshift-multus/network-metrics-daemon-k77fk" Sep 18 06:22:54.164883 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:54.164821 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 18 06:22:54.164883 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:54.164854 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 06:22:54.165019 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:54.164900 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e57e7a14-2ca6-4f41-875f-822739fa1c43-original-pull-secret podName:e57e7a14-2ca6-4f41-875f-822739fa1c43 nodeName:}" failed. No retries permitted until 2026-09-18 06:23:26.164879658 +0000 UTC m=+67.904295815 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e57e7a14-2ca6-4f41-875f-822739fa1c43-original-pull-secret") pod "global-pull-secret-syncer-2jzcv" (UID: "e57e7a14-2ca6-4f41-875f-822739fa1c43") : object "kube-system"/"original-pull-secret" not registered Sep 18 06:22:54.165019 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:54.164920 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66dca255-1289-49ec-b200-e950a30e3d9c-metrics-certs podName:66dca255-1289-49ec-b200-e950a30e3d9c nodeName:}" failed. No retries permitted until 2026-09-18 06:23:26.164909418 +0000 UTC m=+67.904325583 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66dca255-1289-49ec-b200-e950a30e3d9c-metrics-certs") pod "network-metrics-daemon-k77fk" (UID: "66dca255-1289-49ec-b200-e950a30e3d9c") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 06:22:54.617672 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:54.617597 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-104.ec2.internal" event="NodeReady" Sep 18 06:22:54.618131 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:54.617747 2567 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 18 06:22:54.655798 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:54.655769 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-ph66s"] Sep 18 06:22:54.684046 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:54.684020 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qxhxp"] Sep 18 06:22:54.684197 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:54.684150 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ph66s" Sep 18 06:22:54.686820 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:54.686764 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 18 06:22:54.686820 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:54.686791 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 18 06:22:54.689878 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:54.687426 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-sn4gt\"" Sep 18 06:22:54.696712 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:54.696691 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ph66s"] Sep 18 06:22:54.696827 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:54.696721 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qxhxp"] Sep 18 06:22:54.696827 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:54.696784 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qxhxp" Sep 18 06:22:54.698836 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:54.698816 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 18 06:22:54.699138 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:54.699120 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-2l9zb\"" Sep 18 06:22:54.699345 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:54.699327 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 18 06:22:54.699468 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:54.699353 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 18 06:22:54.770483 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:54.770457 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfzcn\" (UniqueName: \"kubernetes.io/projected/8c9f31bd-99bd-4a50-8d28-72dfc56290ae-kube-api-access-sfzcn\") pod \"dns-default-ph66s\" (UID: \"8c9f31bd-99bd-4a50-8d28-72dfc56290ae\") " pod="openshift-dns/dns-default-ph66s" Sep 18 06:22:54.770617 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:54.770491 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c9f31bd-99bd-4a50-8d28-72dfc56290ae-metrics-tls\") pod \"dns-default-ph66s\" (UID: \"8c9f31bd-99bd-4a50-8d28-72dfc56290ae\") " pod="openshift-dns/dns-default-ph66s" Sep 18 06:22:54.770617 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:54.770544 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8c9f31bd-99bd-4a50-8d28-72dfc56290ae-config-volume\") pod \"dns-default-ph66s\" (UID: \"8c9f31bd-99bd-4a50-8d28-72dfc56290ae\") " pod="openshift-dns/dns-default-ph66s" Sep 18 06:22:54.770617 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:54.770572 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c9f31bd-99bd-4a50-8d28-72dfc56290ae-tmp-dir\") pod \"dns-default-ph66s\" (UID: \"8c9f31bd-99bd-4a50-8d28-72dfc56290ae\") " pod="openshift-dns/dns-default-ph66s" Sep 18 06:22:54.871501 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:54.871427 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c9f31bd-99bd-4a50-8d28-72dfc56290ae-tmp-dir\") pod \"dns-default-ph66s\" (UID: \"8c9f31bd-99bd-4a50-8d28-72dfc56290ae\") " pod="openshift-dns/dns-default-ph66s" Sep 18 06:22:54.871501 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:54.871486 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfzcn\" (UniqueName: \"kubernetes.io/projected/8c9f31bd-99bd-4a50-8d28-72dfc56290ae-kube-api-access-sfzcn\") pod \"dns-default-ph66s\" (UID: \"8c9f31bd-99bd-4a50-8d28-72dfc56290ae\") " pod="openshift-dns/dns-default-ph66s" Sep 18 06:22:54.871706 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:54.871611 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c9f31bd-99bd-4a50-8d28-72dfc56290ae-metrics-tls\") pod \"dns-default-ph66s\" (UID: \"8c9f31bd-99bd-4a50-8d28-72dfc56290ae\") " pod="openshift-dns/dns-default-ph66s" Sep 18 06:22:54.871706 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:54.871682 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74ptf\" (UniqueName: \"kubernetes.io/projected/3320df23-25b7-4e16-9024-076401f7ca3e-kube-api-access-74ptf\") pod \"ingress-canary-qxhxp\" (UID: \"3320df23-25b7-4e16-9024-076401f7ca3e\") " pod="openshift-ingress-canary/ingress-canary-qxhxp" Sep 18 06:22:54.871813 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:54.871717 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c9f31bd-99bd-4a50-8d28-72dfc56290ae-tmp-dir\") pod \"dns-default-ph66s\" (UID: \"8c9f31bd-99bd-4a50-8d28-72dfc56290ae\") " pod="openshift-dns/dns-default-ph66s" Sep 18 06:22:54.871813 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:54.871734 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 06:22:54.871813 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:54.871741 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8c9f31bd-99bd-4a50-8d28-72dfc56290ae-config-volume\") pod \"dns-default-ph66s\" (UID: \"8c9f31bd-99bd-4a50-8d28-72dfc56290ae\") " pod="openshift-dns/dns-default-ph66s" Sep 18 06:22:54.871813 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:54.871768 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3320df23-25b7-4e16-9024-076401f7ca3e-cert\") pod \"ingress-canary-qxhxp\" (UID: \"3320df23-25b7-4e16-9024-076401f7ca3e\") " pod="openshift-ingress-canary/ingress-canary-qxhxp" Sep 18 06:22:54.871813 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:54.871789 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c9f31bd-99bd-4a50-8d28-72dfc56290ae-metrics-tls podName:8c9f31bd-99bd-4a50-8d28-72dfc56290ae nodeName:}" failed. No retries permitted until 2026-09-18 06:22:55.371772892 +0000 UTC m=+37.111189047 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8c9f31bd-99bd-4a50-8d28-72dfc56290ae-metrics-tls") pod "dns-default-ph66s" (UID: "8c9f31bd-99bd-4a50-8d28-72dfc56290ae") : secret "dns-default-metrics-tls" not found Sep 18 06:22:54.872170 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:54.872149 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8c9f31bd-99bd-4a50-8d28-72dfc56290ae-config-volume\") pod \"dns-default-ph66s\" (UID: \"8c9f31bd-99bd-4a50-8d28-72dfc56290ae\") " pod="openshift-dns/dns-default-ph66s" Sep 18 06:22:54.881151 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:54.881128 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfzcn\" (UniqueName: \"kubernetes.io/projected/8c9f31bd-99bd-4a50-8d28-72dfc56290ae-kube-api-access-sfzcn\") pod \"dns-default-ph66s\" (UID: \"8c9f31bd-99bd-4a50-8d28-72dfc56290ae\") " pod="openshift-dns/dns-default-ph66s" Sep 18 06:22:54.972100 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:54.972078 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74ptf\" (UniqueName: \"kubernetes.io/projected/3320df23-25b7-4e16-9024-076401f7ca3e-kube-api-access-74ptf\") pod \"ingress-canary-qxhxp\" (UID: \"3320df23-25b7-4e16-9024-076401f7ca3e\") " pod="openshift-ingress-canary/ingress-canary-qxhxp" Sep 18 06:22:54.972204 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:54.972115 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3320df23-25b7-4e16-9024-076401f7ca3e-cert\") pod \"ingress-canary-qxhxp\" (UID: \"3320df23-25b7-4e16-9024-076401f7ca3e\") " pod="openshift-ingress-canary/ingress-canary-qxhxp" Sep 18 06:22:54.974250 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:54.974227 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3320df23-25b7-4e16-9024-076401f7ca3e-cert\") pod \"ingress-canary-qxhxp\" (UID: \"3320df23-25b7-4e16-9024-076401f7ca3e\") " pod="openshift-ingress-canary/ingress-canary-qxhxp" Sep 18 06:22:54.990751 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:54.990731 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74ptf\" (UniqueName: \"kubernetes.io/projected/3320df23-25b7-4e16-9024-076401f7ca3e-kube-api-access-74ptf\") pod \"ingress-canary-qxhxp\" (UID: \"3320df23-25b7-4e16-9024-076401f7ca3e\") " pod="openshift-ingress-canary/ingress-canary-qxhxp" Sep 18 06:22:55.006133 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:55.006115 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qxhxp" Sep 18 06:22:55.301580 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:55.301546 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qxhxp"] Sep 18 06:22:55.304645 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:22:55.304619 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3320df23_25b7_4e16_9024_076401f7ca3e.slice/crio-8cccc8d601338e78c99a4ab4c74612796c1b9d3bdb888fe2c1a02b341baad2e0 WatchSource:0}: Error finding container 8cccc8d601338e78c99a4ab4c74612796c1b9d3bdb888fe2c1a02b341baad2e0: Status 404 returned error can't find the container with id 8cccc8d601338e78c99a4ab4c74612796c1b9d3bdb888fe2c1a02b341baad2e0 Sep 18 06:22:55.374857 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:55.374832 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c9f31bd-99bd-4a50-8d28-72dfc56290ae-metrics-tls\") pod \"dns-default-ph66s\" (UID: \"8c9f31bd-99bd-4a50-8d28-72dfc56290ae\") " pod="openshift-dns/dns-default-ph66s" Sep 18 06:22:55.375006 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:55.374987 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 06:22:55.375062 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:55.375052 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c9f31bd-99bd-4a50-8d28-72dfc56290ae-metrics-tls podName:8c9f31bd-99bd-4a50-8d28-72dfc56290ae nodeName:}" failed. No retries permitted until 2026-09-18 06:22:56.375035849 +0000 UTC m=+38.114452000 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8c9f31bd-99bd-4a50-8d28-72dfc56290ae-metrics-tls") pod "dns-default-ph66s" (UID: "8c9f31bd-99bd-4a50-8d28-72dfc56290ae") : secret "dns-default-metrics-tls" not found Sep 18 06:22:55.439808 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:55.439762 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2jzcv" Sep 18 06:22:55.439808 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:55.439786 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k77fk" Sep 18 06:22:55.439918 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:55.439793 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5rhpx" Sep 18 06:22:55.442685 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:55.442666 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 18 06:22:55.442786 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:55.442774 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kj54z\"" Sep 18 06:22:55.442869 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:55.442852 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 18 06:22:55.442929 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:55.442874 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 18 06:22:55.442977 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:55.442941 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 18 06:22:55.443033 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:55.443014 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fg7cb\"" Sep 18 06:22:55.608921 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:55.608891 2567 generic.go:356] "Generic (PLEG): container finished" podID="0b83c068-4913-4ee5-85ed-17f44dc9f0ca" containerID="bb30fb2ca2f0c0a87f07c75f6c31a9919586dc7490c5caba2301b91166932872" exitCode=0 Sep 18 06:22:55.609072 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:55.608983 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q4gmz" event={"ID":"0b83c068-4913-4ee5-85ed-17f44dc9f0ca","Type":"ContainerDied","Data":"bb30fb2ca2f0c0a87f07c75f6c31a9919586dc7490c5caba2301b91166932872"} Sep 18 06:22:55.610142 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:55.610119 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qxhxp" event={"ID":"3320df23-25b7-4e16-9024-076401f7ca3e","Type":"ContainerStarted","Data":"8cccc8d601338e78c99a4ab4c74612796c1b9d3bdb888fe2c1a02b341baad2e0"} Sep 18 06:22:56.383985 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:56.383952 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c9f31bd-99bd-4a50-8d28-72dfc56290ae-metrics-tls\") pod \"dns-default-ph66s\" (UID: \"8c9f31bd-99bd-4a50-8d28-72dfc56290ae\") " pod="openshift-dns/dns-default-ph66s" Sep 18 06:22:56.384740 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:56.384128 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 06:22:56.384740 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:56.384179 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c9f31bd-99bd-4a50-8d28-72dfc56290ae-metrics-tls podName:8c9f31bd-99bd-4a50-8d28-72dfc56290ae nodeName:}" failed. No retries permitted until 2026-09-18 06:22:58.384165218 +0000 UTC m=+40.123581369 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8c9f31bd-99bd-4a50-8d28-72dfc56290ae-metrics-tls") pod "dns-default-ph66s" (UID: "8c9f31bd-99bd-4a50-8d28-72dfc56290ae") : secret "dns-default-metrics-tls" not found Sep 18 06:22:56.616452 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:56.616190 2567 generic.go:356] "Generic (PLEG): container finished" podID="0b83c068-4913-4ee5-85ed-17f44dc9f0ca" containerID="a3f6d5bf8e6d827024c7dd89ee744e56e89584f05c27e3552387b39597ec2d90" exitCode=0 Sep 18 06:22:56.616620 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:56.616269 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q4gmz" event={"ID":"0b83c068-4913-4ee5-85ed-17f44dc9f0ca","Type":"ContainerDied","Data":"a3f6d5bf8e6d827024c7dd89ee744e56e89584f05c27e3552387b39597ec2d90"} Sep 18 06:22:57.622297 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:57.622262 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q4gmz" event={"ID":"0b83c068-4913-4ee5-85ed-17f44dc9f0ca","Type":"ContainerStarted","Data":"c20d593a267ff558c484be3d3d93cc5b525fe010a25685aa7f628e690fcf48ca"} Sep 18 06:22:57.623741 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:57.623718 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qxhxp" event={"ID":"3320df23-25b7-4e16-9024-076401f7ca3e","Type":"ContainerStarted","Data":"145a3aa7b033dddd0de03ac61dd45b4bcf2c6823f35013ad7a438c52ad0ee114"} Sep 18 06:22:57.643122 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:57.643081 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-q4gmz" podStartSLOduration=8.878806781 podStartE2EDuration="37.64306692s" totalImagesPullingTime="28.764260139s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 06:22:20 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:22:57.641753614 +0000 UTC m=+39.381169786" watchObservedRunningTime="2026-09-18 06:22:57.64306692 +0000 UTC m=+39.382483094" Sep 18 06:22:57.655332 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:57.655288 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qxhxp" podStartSLOduration=1.630152582 podStartE2EDuration="3.655275125s" totalImagesPullingTime="2.025122543s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:22:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:22:57.654550851 +0000 UTC m=+39.393967033" watchObservedRunningTime="2026-09-18 06:22:57.655275125 +0000 UTC m=+39.394691343" Sep 18 06:22:58.398070 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:22:58.398037 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c9f31bd-99bd-4a50-8d28-72dfc56290ae-metrics-tls\") pod \"dns-default-ph66s\" (UID: \"8c9f31bd-99bd-4a50-8d28-72dfc56290ae\") " pod="openshift-dns/dns-default-ph66s" Sep 18 06:22:58.398234 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:58.398156 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 06:22:58.398234 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:22:58.398205 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c9f31bd-99bd-4a50-8d28-72dfc56290ae-metrics-tls podName:8c9f31bd-99bd-4a50-8d28-72dfc56290ae nodeName:}" failed. No retries permitted until 2026-09-18 06:23:02.398192633 +0000 UTC m=+44.137608798 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8c9f31bd-99bd-4a50-8d28-72dfc56290ae-metrics-tls") pod "dns-default-ph66s" (UID: "8c9f31bd-99bd-4a50-8d28-72dfc56290ae") : secret "dns-default-metrics-tls" not found Sep 18 06:23:02.424504 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:23:02.424472 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c9f31bd-99bd-4a50-8d28-72dfc56290ae-metrics-tls\") pod \"dns-default-ph66s\" (UID: \"8c9f31bd-99bd-4a50-8d28-72dfc56290ae\") " pod="openshift-dns/dns-default-ph66s" Sep 18 06:23:02.424865 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:23:02.424574 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 06:23:02.424865 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:23:02.424623 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c9f31bd-99bd-4a50-8d28-72dfc56290ae-metrics-tls podName:8c9f31bd-99bd-4a50-8d28-72dfc56290ae nodeName:}" failed. No retries permitted until 2026-09-18 06:23:10.424610161 +0000 UTC m=+52.164026313 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8c9f31bd-99bd-4a50-8d28-72dfc56290ae-metrics-tls") pod "dns-default-ph66s" (UID: "8c9f31bd-99bd-4a50-8d28-72dfc56290ae") : secret "dns-default-metrics-tls" not found Sep 18 06:23:10.475874 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:23:10.475845 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c9f31bd-99bd-4a50-8d28-72dfc56290ae-metrics-tls\") pod \"dns-default-ph66s\" (UID: \"8c9f31bd-99bd-4a50-8d28-72dfc56290ae\") " pod="openshift-dns/dns-default-ph66s" Sep 18 06:23:10.476247 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:23:10.475975 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 06:23:10.476247 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:23:10.476021 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c9f31bd-99bd-4a50-8d28-72dfc56290ae-metrics-tls podName:8c9f31bd-99bd-4a50-8d28-72dfc56290ae nodeName:}" failed. No retries permitted until 2026-09-18 06:23:26.476009221 +0000 UTC m=+68.215425374 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8c9f31bd-99bd-4a50-8d28-72dfc56290ae-metrics-tls") pod "dns-default-ph66s" (UID: "8c9f31bd-99bd-4a50-8d28-72dfc56290ae") : secret "dns-default-metrics-tls" not found Sep 18 06:23:19.599515 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:23:19.599487 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hjr4h" Sep 18 06:23:26.087065 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:23:26.087010 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwmtn\" (UniqueName: \"kubernetes.io/projected/ce5f0b70-0633-4019-a805-85d798616ac2-kube-api-access-pwmtn\") pod \"network-check-target-5rhpx\" (UID: \"ce5f0b70-0633-4019-a805-85d798616ac2\") " pod="openshift-network-diagnostics/network-check-target-5rhpx" Sep 18 06:23:26.089949 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:23:26.089931 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 18 06:23:26.099943 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:23:26.099920 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 18 06:23:26.111845 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:23:26.111826 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwmtn\" (UniqueName: \"kubernetes.io/projected/ce5f0b70-0633-4019-a805-85d798616ac2-kube-api-access-pwmtn\") pod \"network-check-target-5rhpx\" (UID: \"ce5f0b70-0633-4019-a805-85d798616ac2\") " pod="openshift-network-diagnostics/network-check-target-5rhpx" Sep 18 06:23:26.187515 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:23:26.187480 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66dca255-1289-49ec-b200-e950a30e3d9c-metrics-certs\") pod \"network-metrics-daemon-k77fk\" (UID: \"66dca255-1289-49ec-b200-e950a30e3d9c\") " pod="openshift-multus/network-metrics-daemon-k77fk" Sep 18 06:23:26.187620 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:23:26.187532 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e57e7a14-2ca6-4f41-875f-822739fa1c43-original-pull-secret\") pod \"global-pull-secret-syncer-2jzcv\" (UID: \"e57e7a14-2ca6-4f41-875f-822739fa1c43\") " pod="kube-system/global-pull-secret-syncer-2jzcv" Sep 18 06:23:26.189894 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:23:26.189879 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 18 06:23:26.189948 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:23:26.189917 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 18 06:23:26.198004 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:23:26.197985 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 18 06:23:26.198094 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:23:26.198040 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66dca255-1289-49ec-b200-e950a30e3d9c-metrics-certs podName:66dca255-1289-49ec-b200-e950a30e3d9c nodeName:}" failed. No retries permitted until 2026-09-18 06:24:30.198025919 +0000 UTC m=+131.937442076 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66dca255-1289-49ec-b200-e950a30e3d9c-metrics-certs") pod "network-metrics-daemon-k77fk" (UID: "66dca255-1289-49ec-b200-e950a30e3d9c") : secret "metrics-daemon-secret" not found Sep 18 06:23:26.199916 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:23:26.199899 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e57e7a14-2ca6-4f41-875f-822739fa1c43-original-pull-secret\") pod \"global-pull-secret-syncer-2jzcv\" (UID: \"e57e7a14-2ca6-4f41-875f-822739fa1c43\") " pod="kube-system/global-pull-secret-syncer-2jzcv" Sep 18 06:23:26.348947 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:23:26.348893 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2jzcv" Sep 18 06:23:26.362077 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:23:26.362056 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kj54z\"" Sep 18 06:23:26.370060 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:23:26.370041 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5rhpx" Sep 18 06:23:26.489440 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:23:26.489416 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c9f31bd-99bd-4a50-8d28-72dfc56290ae-metrics-tls\") pod \"dns-default-ph66s\" (UID: \"8c9f31bd-99bd-4a50-8d28-72dfc56290ae\") " pod="openshift-dns/dns-default-ph66s" Sep 18 06:23:26.489581 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:23:26.489562 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 06:23:26.489637 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:23:26.489627 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c9f31bd-99bd-4a50-8d28-72dfc56290ae-metrics-tls podName:8c9f31bd-99bd-4a50-8d28-72dfc56290ae nodeName:}" failed. No retries permitted until 2026-09-18 06:23:58.489610797 +0000 UTC m=+100.229026949 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8c9f31bd-99bd-4a50-8d28-72dfc56290ae-metrics-tls") pod "dns-default-ph66s" (UID: "8c9f31bd-99bd-4a50-8d28-72dfc56290ae") : secret "dns-default-metrics-tls" not found Sep 18 06:23:26.616070 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:23:26.616013 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2jzcv"] Sep 18 06:23:26.619115 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:23:26.619082 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode57e7a14_2ca6_4f41_875f_822739fa1c43.slice/crio-a7da86c232a76d80e46997b7a29f690096829860fa1204c8916888e9c835b358 WatchSource:0}: Error finding container a7da86c232a76d80e46997b7a29f690096829860fa1204c8916888e9c835b358: Status 404 returned error can't find the container with id a7da86c232a76d80e46997b7a29f690096829860fa1204c8916888e9c835b358 Sep 18 06:23:26.646680 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:23:26.646654 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5rhpx"] Sep 18 06:23:26.650400 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:23:26.650361 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce5f0b70_0633_4019_a805_85d798616ac2.slice/crio-25bb48450d88856fba0c4f5c9f57aa0d2ea5c70402b5e0f4928bba6343e1a821 WatchSource:0}: Error finding container 25bb48450d88856fba0c4f5c9f57aa0d2ea5c70402b5e0f4928bba6343e1a821: Status 404 returned error can't find the container with id 25bb48450d88856fba0c4f5c9f57aa0d2ea5c70402b5e0f4928bba6343e1a821 Sep 18 06:23:26.693474 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:23:26.693448 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5rhpx" event={"ID":"ce5f0b70-0633-4019-a805-85d798616ac2","Type":"ContainerStarted","Data":"25bb48450d88856fba0c4f5c9f57aa0d2ea5c70402b5e0f4928bba6343e1a821"} Sep 18 06:23:26.694583 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:23:26.694560 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2jzcv" event={"ID":"e57e7a14-2ca6-4f41-875f-822739fa1c43","Type":"ContainerStarted","Data":"a7da86c232a76d80e46997b7a29f690096829860fa1204c8916888e9c835b358"} Sep 18 06:23:31.709721 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:23:31.709668 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2jzcv" event={"ID":"e57e7a14-2ca6-4f41-875f-822739fa1c43","Type":"ContainerStarted","Data":"409f3ae6ac8260d3be700b11eca84c622a92805201b978cc3180fb465eaffee4"} Sep 18 06:23:31.711066 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:23:31.711043 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5rhpx" event={"ID":"ce5f0b70-0633-4019-a805-85d798616ac2","Type":"ContainerStarted","Data":"4fcb7376bc9840300aaa21feceb40347e25154d0f6e7ce31c689da933ecdb91e"} Sep 18 06:23:31.711182 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:23:31.711168 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5rhpx" Sep 18 06:23:31.725093 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:23:31.725033 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-2jzcv" podStartSLOduration=67.433489093 podStartE2EDuration="1m11.725016316s" totalImagesPullingTime="4.291527223s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 06:22:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:23:31.724276194 +0000 UTC m=+73.463692368" watchObservedRunningTime="2026-09-18 06:23:31.725016316 +0000 UTC m=+73.464432492" Sep 18 06:23:31.738865 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:23:31.738829 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5rhpx" podStartSLOduration=67.47913199 podStartE2EDuration="1m11.738815697s" totalImagesPullingTime="4.259683707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:22:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:23:31.737954753 +0000 UTC m=+73.477370947" watchObservedRunningTime="2026-09-18 06:23:31.738815697 +0000 UTC m=+73.478231862" Sep 18 06:23:33.912494 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:23:33.912464 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d67655786-ldvcr"] Sep 18 06:23:33.914596 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:23:33.914545 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d67655786-ldvcr" Sep 18 06:23:33.917228 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:23:33.917207 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Sep 18 06:23:33.918707 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:23:33.918689 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 18 06:23:33.918806 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:23:33.918715 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 18 06:23:33.918806 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:23:33.918777 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 18 06:23:33.918930 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:23:33.918820 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-hsg77\"" Sep 18 06:23:33.925196 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:23:33.925177 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d67655786-ldvcr"] Sep 18 06:23:33.934124 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:23:33.934106 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/97bae500-dbf0-4bce-b633-5f888d322515-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5d67655786-ldvcr\" (UID: \"97bae500-dbf0-4bce-b633-5f888d322515\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d67655786-ldvcr" Sep 18 06:23:33.934211 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:23:33.934145 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvfv8\" (UniqueName: \"kubernetes.io/projected/97bae500-dbf0-4bce-b633-5f888d322515-kube-api-access-qvfv8\") pod \"managed-serviceaccount-addon-agent-5d67655786-ldvcr\" (UID: \"97bae500-dbf0-4bce-b633-5f888d322515\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d67655786-ldvcr" Sep 18 06:23:34.034706 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:23:34.034666 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/97bae500-dbf0-4bce-b633-5f888d322515-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5d67655786-ldvcr\" (UID: \"97bae500-dbf0-4bce-b633-5f888d322515\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d67655786-ldvcr" Sep 18 06:23:34.034890 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:23:34.034737 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvfv8\" (UniqueName: \"kubernetes.io/projected/97bae500-dbf0-4bce-b633-5f888d322515-kube-api-access-qvfv8\") pod \"managed-serviceaccount-addon-agent-5d67655786-ldvcr\" (UID: \"97bae500-dbf0-4bce-b633-5f888d322515\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d67655786-ldvcr" Sep 18 06:23:34.037065 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:23:34.037046 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/97bae500-dbf0-4bce-b633-5f888d322515-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5d67655786-ldvcr\" (UID: \"97bae500-dbf0-4bce-b633-5f888d322515\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d67655786-ldvcr" Sep 18 06:23:34.042312 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:23:34.042289 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvfv8\" (UniqueName: \"kubernetes.io/projected/97bae500-dbf0-4bce-b633-5f888d322515-kube-api-access-qvfv8\") pod \"managed-serviceaccount-addon-agent-5d67655786-ldvcr\" (UID: \"97bae500-dbf0-4bce-b633-5f888d322515\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d67655786-ldvcr" Sep 18 06:23:34.232707 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:23:34.232649 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d67655786-ldvcr" Sep 18 06:23:34.496972 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:23:34.496914 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d67655786-ldvcr"] Sep 18 06:23:34.501015 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:23:34.500992 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97bae500_dbf0_4bce_b633_5f888d322515.slice/crio-08059ca326dd18939f5c6e44e93474b4710ec71c16bab0aeb2bced1518f2f77e WatchSource:0}: Error finding container 08059ca326dd18939f5c6e44e93474b4710ec71c16bab0aeb2bced1518f2f77e: Status 404 returned error can't find the container with id 08059ca326dd18939f5c6e44e93474b4710ec71c16bab0aeb2bced1518f2f77e Sep 18 06:23:34.719574 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:23:34.719544 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d67655786-ldvcr" event={"ID":"97bae500-dbf0-4bce-b633-5f888d322515","Type":"ContainerStarted","Data":"08059ca326dd18939f5c6e44e93474b4710ec71c16bab0aeb2bced1518f2f77e"} Sep 18 06:23:37.727651 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:23:37.727612 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d67655786-ldvcr" event={"ID":"97bae500-dbf0-4bce-b633-5f888d322515","Type":"ContainerStarted","Data":"609033f0cf13c98ba0904440b2f9d74c7363f32c031f772ebd83813e09f0be86"} Sep 18 06:23:37.741386 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:23:37.741333 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d67655786-ldvcr" podStartSLOduration=2.360324283 podStartE2EDuration="4.741320956s" totalImagesPullingTime="2.380996673s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:23:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:23:37.740798111 +0000 UTC m=+79.480214284" watchObservedRunningTime="2026-09-18 06:23:37.741320956 +0000 UTC m=+79.480737129" Sep 18 06:23:58.588498 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:23:58.588454 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c9f31bd-99bd-4a50-8d28-72dfc56290ae-metrics-tls\") pod \"dns-default-ph66s\" (UID: \"8c9f31bd-99bd-4a50-8d28-72dfc56290ae\") " pod="openshift-dns/dns-default-ph66s" Sep 18 06:23:58.588865 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:23:58.588559 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 06:23:58.588865 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:23:58.588618 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c9f31bd-99bd-4a50-8d28-72dfc56290ae-metrics-tls podName:8c9f31bd-99bd-4a50-8d28-72dfc56290ae nodeName:}" failed. No retries permitted until 2026-09-18 06:25:02.588603279 +0000 UTC m=+164.328019431 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8c9f31bd-99bd-4a50-8d28-72dfc56290ae-metrics-tls") pod "dns-default-ph66s" (UID: "8c9f31bd-99bd-4a50-8d28-72dfc56290ae") : secret "dns-default-metrics-tls" not found Sep 18 06:24:02.715571 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:24:02.715545 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5rhpx" Sep 18 06:24:30.287482 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:24:30.287442 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66dca255-1289-49ec-b200-e950a30e3d9c-metrics-certs\") pod \"network-metrics-daemon-k77fk\" (UID: \"66dca255-1289-49ec-b200-e950a30e3d9c\") " pod="openshift-multus/network-metrics-daemon-k77fk" Sep 18 06:24:30.287892 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:24:30.287553 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 18 06:24:30.287892 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:24:30.287608 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66dca255-1289-49ec-b200-e950a30e3d9c-metrics-certs podName:66dca255-1289-49ec-b200-e950a30e3d9c nodeName:}" failed. No retries permitted until 2026-09-18 06:26:32.287594005 +0000 UTC m=+254.027010171 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66dca255-1289-49ec-b200-e950a30e3d9c-metrics-certs") pod "network-metrics-daemon-k77fk" (UID: "66dca255-1289-49ec-b200-e950a30e3d9c") : secret "metrics-daemon-secret" not found Sep 18 06:24:57.698102 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:24:57.698011 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-ph66s" podUID="8c9f31bd-99bd-4a50-8d28-72dfc56290ae" Sep 18 06:24:57.915455 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:24:57.915421 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ph66s" Sep 18 06:24:58.454346 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:24:58.454314 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-k77fk" podUID="66dca255-1289-49ec-b200-e950a30e3d9c" Sep 18 06:25:02.590451 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:25:02.590378 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c9f31bd-99bd-4a50-8d28-72dfc56290ae-metrics-tls\") pod \"dns-default-ph66s\" (UID: \"8c9f31bd-99bd-4a50-8d28-72dfc56290ae\") " pod="openshift-dns/dns-default-ph66s" Sep 18 06:25:02.590816 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:25:02.590496 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 06:25:02.590816 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:25:02.590568 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c9f31bd-99bd-4a50-8d28-72dfc56290ae-metrics-tls podName:8c9f31bd-99bd-4a50-8d28-72dfc56290ae nodeName:}" failed. No retries permitted until 2026-09-18 06:27:04.590550595 +0000 UTC m=+286.329966750 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8c9f31bd-99bd-4a50-8d28-72dfc56290ae-metrics-tls") pod "dns-default-ph66s" (UID: "8c9f31bd-99bd-4a50-8d28-72dfc56290ae") : secret "dns-default-metrics-tls" not found Sep 18 06:25:11.439684 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:25:11.439633 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k77fk" Sep 18 06:25:37.011438 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:25:37.011411 2567 generic.go:356] "Generic (PLEG): container finished" podID="97bae500-dbf0-4bce-b633-5f888d322515" containerID="609033f0cf13c98ba0904440b2f9d74c7363f32c031f772ebd83813e09f0be86" exitCode=255 Sep 18 06:25:37.011773 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:25:37.011468 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d67655786-ldvcr" event={"ID":"97bae500-dbf0-4bce-b633-5f888d322515","Type":"ContainerDied","Data":"609033f0cf13c98ba0904440b2f9d74c7363f32c031f772ebd83813e09f0be86"} Sep 18 06:25:37.011773 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:25:37.011732 2567 scope.go:117] "RemoveContainer" containerID="609033f0cf13c98ba0904440b2f9d74c7363f32c031f772ebd83813e09f0be86" Sep 18 06:25:38.016177 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:25:38.016145 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d67655786-ldvcr" event={"ID":"97bae500-dbf0-4bce-b633-5f888d322515","Type":"ContainerStarted","Data":"f513789d539239b1170af437cf5beba7d1b467cac84ad54c1281ae263601e57e"} Sep 18 06:26:26.907299 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:26.907266 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-qcsfv"] Sep 18 06:26:26.909070 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:26.909054 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-qcsfv" Sep 18 06:26:26.910808 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:26.910766 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-tqkh6"] Sep 18 06:26:26.912695 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:26.912645 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-2s49m\"" Sep 18 06:26:26.914172 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:26.912944 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-5bn97"] Sep 18 06:26:26.914172 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:26.913206 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 18 06:26:26.914172 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:26.913228 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 18 06:26:26.914172 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:26.914097 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-tqkh6" Sep 18 06:26:26.914172 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:26.914172 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 18 06:26:26.916904 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:26.916887 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-4hlk2\"" Sep 18 06:26:26.917809 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:26.917791 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-75b6d7b44b-fvj8r"] Sep 18 06:26:26.917895 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:26.917845 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 18 06:26:26.917953 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:26.917919 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-5bn97" Sep 18 06:26:26.919163 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:26.919144 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 18 06:26:26.919260 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:26.919180 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 18 06:26:26.919260 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:26.919250 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 18 06:26:26.919903 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:26.919884 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-75b6d7b44b-fvj8r" Sep 18 06:26:26.920157 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:26.920137 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-85xt2\"" Sep 18 06:26:26.920418 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:26.920399 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 18 06:26:26.920567 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:26.920552 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 18 06:26:26.920610 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:26.920574 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 18 06:26:26.920610 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:26.920585 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 18 06:26:26.923105 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:26.923087 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Sep 18 06:26:26.923258 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:26.923241 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-5kbpd\"" Sep 18 06:26:26.923388 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:26.923354 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Sep 18 06:26:26.923675 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:26.923656 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Sep 18 06:26:26.923776 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:26.923678 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 18 06:26:26.923776 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:26.923748 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Sep 18 06:26:26.924166 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:26.924146 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-qcsfv"] Sep 18 06:26:26.924598 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:26.924579 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 18 06:26:26.926578 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:26.926561 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-tqkh6"] Sep 18 06:26:26.927592 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:26.927567 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 18 06:26:26.931175 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:26.931145 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-5bn97"] Sep 18 06:26:26.932900 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:26.932854 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-75b6d7b44b-fvj8r"] Sep 18 06:26:27.001419 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.001388 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-kzkk8"] Sep 18 06:26:27.003494 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.003406 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-kzkk8" Sep 18 06:26:27.004236 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.004217 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-fthpb"] Sep 18 06:26:27.005684 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.005663 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 18 06:26:27.005794 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.005704 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 18 06:26:27.005794 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.005759 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 18 06:26:27.005794 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.005769 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-qwsj6\"" Sep 18 06:26:27.005995 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.005980 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 18 06:26:27.006113 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.006097 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-fthpb" Sep 18 06:26:27.008315 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.008295 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-zdbtb\"" Sep 18 06:26:27.008432 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.008321 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 18 06:26:27.008432 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.008322 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 18 06:26:27.009024 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.009009 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7cfd7d874-4m95j"] Sep 18 06:26:27.010943 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.010926 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cfd7d874-4m95j" Sep 18 06:26:27.013116 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.013101 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-d7llw\"" Sep 18 06:26:27.013215 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.013192 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 18 06:26:27.013416 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.013400 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 18 06:26:27.013498 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.013484 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 18 06:26:27.018844 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.018826 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 18 06:26:27.018980 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.018959 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-kzkk8"] Sep 18 06:26:27.020287 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.020266 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-fthpb"] Sep 18 06:26:27.027834 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.027812 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7cfd7d874-4m95j"] Sep 18 06:26:27.036382 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.036343 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b856fd0d-7ea9-4899-958a-98efec61fe44-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-kzkk8\" (UID: \"b856fd0d-7ea9-4899-958a-98efec61fe44\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-kzkk8" Sep 18 06:26:27.036484 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.036389 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dcc10422-179e-4253-95a4-b8621790f350-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-qcsfv\" (UID: \"dcc10422-179e-4253-95a4-b8621790f350\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-qcsfv" Sep 18 06:26:27.036484 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.036459 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b856fd0d-7ea9-4899-958a-98efec61fe44-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-kzkk8\" (UID: \"b856fd0d-7ea9-4899-958a-98efec61fe44\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-kzkk8" Sep 18 06:26:27.036599 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.036489 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee4fa404-921d-48d8-a26d-fa5c15425928-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-5bn97\" (UID: \"ee4fa404-921d-48d8-a26d-fa5c15425928\") " pod="openshift-insights/insights-operator-6f9c644dd9-5bn97" Sep 18 06:26:27.036599 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.036525 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j54rs\" (UniqueName: \"kubernetes.io/projected/dcc10422-179e-4253-95a4-b8621790f350-kube-api-access-j54rs\") pod \"cluster-samples-operator-685d7d68b5-qcsfv\" (UID: \"dcc10422-179e-4253-95a4-b8621790f350\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-qcsfv" Sep 18 06:26:27.036599 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.036547 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e16d183f-ea45-4a4c-94be-f078321c3c79-default-certificate\") pod \"router-default-75b6d7b44b-fvj8r\" (UID: \"e16d183f-ea45-4a4c-94be-f078321c3c79\") " pod="openshift-ingress/router-default-75b6d7b44b-fvj8r" Sep 18 06:26:27.036599 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.036569 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee4fa404-921d-48d8-a26d-fa5c15425928-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-5bn97\" (UID: \"ee4fa404-921d-48d8-a26d-fa5c15425928\") " pod="openshift-insights/insights-operator-6f9c644dd9-5bn97" Sep 18 06:26:27.036599 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.036595 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e16d183f-ea45-4a4c-94be-f078321c3c79-service-ca-bundle\") pod \"router-default-75b6d7b44b-fvj8r\" (UID: \"e16d183f-ea45-4a4c-94be-f078321c3c79\") " pod="openshift-ingress/router-default-75b6d7b44b-fvj8r" Sep 18 06:26:27.036777 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.036613 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e16d183f-ea45-4a4c-94be-f078321c3c79-stats-auth\") pod \"router-default-75b6d7b44b-fvj8r\" (UID: \"e16d183f-ea45-4a4c-94be-f078321c3c79\") " pod="openshift-ingress/router-default-75b6d7b44b-fvj8r" Sep 18 06:26:27.036777 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.036641 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ee4fa404-921d-48d8-a26d-fa5c15425928-serving-cert\") pod \"insights-operator-6f9c644dd9-5bn97\" (UID: \"ee4fa404-921d-48d8-a26d-fa5c15425928\") " pod="openshift-insights/insights-operator-6f9c644dd9-5bn97" Sep 18 06:26:27.036777 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.036678 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c26181ee-5e97-4934-a7da-71a24e303856-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-tqkh6\" (UID: \"c26181ee-5e97-4934-a7da-71a24e303856\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-tqkh6" Sep 18 06:26:27.036777 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.036701 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zngf6\" (UniqueName: \"kubernetes.io/projected/6164b435-ee87-4137-b551-f951321399e6-kube-api-access-zngf6\") pod \"volume-data-source-validator-5c4db8bdf9-fthpb\" (UID: \"6164b435-ee87-4137-b551-f951321399e6\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-fthpb" Sep 18 06:26:27.036777 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.036724 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ee4fa404-921d-48d8-a26d-fa5c15425928-snapshots\") pod \"insights-operator-6f9c644dd9-5bn97\" (UID: \"ee4fa404-921d-48d8-a26d-fa5c15425928\") " pod="openshift-insights/insights-operator-6f9c644dd9-5bn97" Sep 18 06:26:27.036777 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.036764 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np696\" (UniqueName: \"kubernetes.io/projected/ee4fa404-921d-48d8-a26d-fa5c15425928-kube-api-access-np696\") pod \"insights-operator-6f9c644dd9-5bn97\" (UID: \"ee4fa404-921d-48d8-a26d-fa5c15425928\") " pod="openshift-insights/insights-operator-6f9c644dd9-5bn97" Sep 18 06:26:27.036989 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.036786 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c26181ee-5e97-4934-a7da-71a24e303856-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-tqkh6\" (UID: \"c26181ee-5e97-4934-a7da-71a24e303856\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-tqkh6" Sep 18 06:26:27.036989 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.036805 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfprn\" (UniqueName: \"kubernetes.io/projected/c26181ee-5e97-4934-a7da-71a24e303856-kube-api-access-pfprn\") pod \"cluster-monitoring-operator-847b8bd8d7-tqkh6\" (UID: \"c26181ee-5e97-4934-a7da-71a24e303856\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-tqkh6" Sep 18 06:26:27.036989 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.036835 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg25l\" (UniqueName: \"kubernetes.io/projected/b856fd0d-7ea9-4899-958a-98efec61fe44-kube-api-access-cg25l\") pod \"kube-storage-version-migrator-operator-55889f7b47-kzkk8\" (UID: \"b856fd0d-7ea9-4899-958a-98efec61fe44\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-kzkk8" Sep 18 06:26:27.036989 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.036851 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e16d183f-ea45-4a4c-94be-f078321c3c79-metrics-certs\") pod \"router-default-75b6d7b44b-fvj8r\" (UID: \"e16d183f-ea45-4a4c-94be-f078321c3c79\") " pod="openshift-ingress/router-default-75b6d7b44b-fvj8r" Sep 18 06:26:27.036989 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.036876 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzdhh\" (UniqueName: \"kubernetes.io/projected/e16d183f-ea45-4a4c-94be-f078321c3c79-kube-api-access-dzdhh\") pod \"router-default-75b6d7b44b-fvj8r\" (UID: \"e16d183f-ea45-4a4c-94be-f078321c3c79\") " pod="openshift-ingress/router-default-75b6d7b44b-fvj8r" Sep 18 06:26:27.036989 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.036899 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee4fa404-921d-48d8-a26d-fa5c15425928-tmp\") pod \"insights-operator-6f9c644dd9-5bn97\" (UID: \"ee4fa404-921d-48d8-a26d-fa5c15425928\") " pod="openshift-insights/insights-operator-6f9c644dd9-5bn97" Sep 18 06:26:27.137251 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.137220 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b856fd0d-7ea9-4899-958a-98efec61fe44-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-kzkk8\" (UID: \"b856fd0d-7ea9-4899-958a-98efec61fe44\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-kzkk8" Sep 18 06:26:27.137251 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.137255 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dcc10422-179e-4253-95a4-b8621790f350-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-qcsfv\" (UID: \"dcc10422-179e-4253-95a4-b8621790f350\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-qcsfv" Sep 18 06:26:27.137522 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.137272 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg25l\" (UniqueName: \"kubernetes.io/projected/b856fd0d-7ea9-4899-958a-98efec61fe44-kube-api-access-cg25l\") pod \"kube-storage-version-migrator-operator-55889f7b47-kzkk8\" (UID: \"b856fd0d-7ea9-4899-958a-98efec61fe44\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-kzkk8" Sep 18 06:26:27.137522 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.137290 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e16d183f-ea45-4a4c-94be-f078321c3c79-metrics-certs\") pod \"router-default-75b6d7b44b-fvj8r\" (UID: \"e16d183f-ea45-4a4c-94be-f078321c3c79\") " pod="openshift-ingress/router-default-75b6d7b44b-fvj8r" Sep 18 06:26:27.137522 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.137313 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee4fa404-921d-48d8-a26d-fa5c15425928-tmp\") pod \"insights-operator-6f9c644dd9-5bn97\" (UID: \"ee4fa404-921d-48d8-a26d-fa5c15425928\") " pod="openshift-insights/insights-operator-6f9c644dd9-5bn97" Sep 18 06:26:27.137522 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.137337 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzdhh\" (UniqueName: \"kubernetes.io/projected/e16d183f-ea45-4a4c-94be-f078321c3c79-kube-api-access-dzdhh\") pod \"router-default-75b6d7b44b-fvj8r\" (UID: \"e16d183f-ea45-4a4c-94be-f078321c3c79\") " pod="openshift-ingress/router-default-75b6d7b44b-fvj8r" Sep 18 06:26:27.137522 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.137383 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np696\" (UniqueName: \"kubernetes.io/projected/ee4fa404-921d-48d8-a26d-fa5c15425928-kube-api-access-np696\") pod \"insights-operator-6f9c644dd9-5bn97\" (UID: \"ee4fa404-921d-48d8-a26d-fa5c15425928\") " pod="openshift-insights/insights-operator-6f9c644dd9-5bn97" Sep 18 06:26:27.137522 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:27.137402 2567 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 18 06:26:27.137522 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.137424 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e16d183f-ea45-4a4c-94be-f078321c3c79-default-certificate\") pod \"router-default-75b6d7b44b-fvj8r\" (UID: \"e16d183f-ea45-4a4c-94be-f078321c3c79\") " pod="openshift-ingress/router-default-75b6d7b44b-fvj8r" Sep 18 06:26:27.137522 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.137445 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee4fa404-921d-48d8-a26d-fa5c15425928-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-5bn97\" (UID: \"ee4fa404-921d-48d8-a26d-fa5c15425928\") " pod="openshift-insights/insights-operator-6f9c644dd9-5bn97" Sep 18 06:26:27.137522 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:27.137462 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e16d183f-ea45-4a4c-94be-f078321c3c79-metrics-certs podName:e16d183f-ea45-4a4c-94be-f078321c3c79 nodeName:}" failed. No retries permitted until 2026-09-18 06:26:27.637445773 +0000 UTC m=+249.376861924 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e16d183f-ea45-4a4c-94be-f078321c3c79-metrics-certs") pod "router-default-75b6d7b44b-fvj8r" (UID: "e16d183f-ea45-4a4c-94be-f078321c3c79") : secret "router-metrics-certs-default" not found Sep 18 06:26:27.137522 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:27.137402 2567 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 18 06:26:27.137522 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:27.137521 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcc10422-179e-4253-95a4-b8621790f350-samples-operator-tls podName:dcc10422-179e-4253-95a4-b8621790f350 nodeName:}" failed. No retries permitted until 2026-09-18 06:26:27.637510616 +0000 UTC m=+249.376926767 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/dcc10422-179e-4253-95a4-b8621790f350-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-qcsfv" (UID: "dcc10422-179e-4253-95a4-b8621790f350") : secret "samples-operator-tls" not found Sep 18 06:26:27.138045 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.137652 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b-registry-tls\") pod \"image-registry-7cfd7d874-4m95j\" (UID: \"ca057b3a-aa6d-45ff-bad8-6dc74bf2009b\") " pod="openshift-image-registry/image-registry-7cfd7d874-4m95j" Sep 18 06:26:27.138045 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.137678 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b-installation-pull-secrets\") pod \"image-registry-7cfd7d874-4m95j\" (UID: \"ca057b3a-aa6d-45ff-bad8-6dc74bf2009b\") " pod="openshift-image-registry/image-registry-7cfd7d874-4m95j" Sep 18 06:26:27.138045 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.137715 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b-image-registry-private-configuration\") pod \"image-registry-7cfd7d874-4m95j\" (UID: \"ca057b3a-aa6d-45ff-bad8-6dc74bf2009b\") " pod="openshift-image-registry/image-registry-7cfd7d874-4m95j" Sep 18 06:26:27.138045 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.137746 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b-ca-trust-extracted\") pod \"image-registry-7cfd7d874-4m95j\" (UID: \"ca057b3a-aa6d-45ff-bad8-6dc74bf2009b\") " pod="openshift-image-registry/image-registry-7cfd7d874-4m95j" Sep 18 06:26:27.138045 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.137777 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee4fa404-921d-48d8-a26d-fa5c15425928-tmp\") pod \"insights-operator-6f9c644dd9-5bn97\" (UID: \"ee4fa404-921d-48d8-a26d-fa5c15425928\") " pod="openshift-insights/insights-operator-6f9c644dd9-5bn97" Sep 18 06:26:27.138045 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.137796 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e16d183f-ea45-4a4c-94be-f078321c3c79-service-ca-bundle\") pod \"router-default-75b6d7b44b-fvj8r\" (UID: \"e16d183f-ea45-4a4c-94be-f078321c3c79\") " pod="openshift-ingress/router-default-75b6d7b44b-fvj8r" Sep 18 06:26:27.138045 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.137834 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e16d183f-ea45-4a4c-94be-f078321c3c79-stats-auth\") pod \"router-default-75b6d7b44b-fvj8r\" (UID: \"e16d183f-ea45-4a4c-94be-f078321c3c79\") " pod="openshift-ingress/router-default-75b6d7b44b-fvj8r" Sep 18 06:26:27.138045 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.137868 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ee4fa404-921d-48d8-a26d-fa5c15425928-serving-cert\") pod \"insights-operator-6f9c644dd9-5bn97\" (UID: \"ee4fa404-921d-48d8-a26d-fa5c15425928\") " pod="openshift-insights/insights-operator-6f9c644dd9-5bn97" Sep 18 06:26:27.138045 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:27.137877 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e16d183f-ea45-4a4c-94be-f078321c3c79-service-ca-bundle podName:e16d183f-ea45-4a4c-94be-f078321c3c79 nodeName:}" failed. No retries permitted until 2026-09-18 06:26:27.637861205 +0000 UTC m=+249.377277358 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e16d183f-ea45-4a4c-94be-f078321c3c79-service-ca-bundle") pod "router-default-75b6d7b44b-fvj8r" (UID: "e16d183f-ea45-4a4c-94be-f078321c3c79") : configmap references non-existent config key: service-ca.crt Sep 18 06:26:27.138045 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.137927 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c26181ee-5e97-4934-a7da-71a24e303856-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-tqkh6\" (UID: \"c26181ee-5e97-4934-a7da-71a24e303856\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-tqkh6" Sep 18 06:26:27.138045 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.137963 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c26181ee-5e97-4934-a7da-71a24e303856-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-tqkh6\" (UID: \"c26181ee-5e97-4934-a7da-71a24e303856\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-tqkh6" Sep 18 06:26:27.138045 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.137992 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfprn\" (UniqueName: \"kubernetes.io/projected/c26181ee-5e97-4934-a7da-71a24e303856-kube-api-access-pfprn\") pod \"cluster-monitoring-operator-847b8bd8d7-tqkh6\" (UID: \"c26181ee-5e97-4934-a7da-71a24e303856\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-tqkh6" Sep 18 06:26:27.138645 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:27.138107 2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 18 06:26:27.138645 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:27.138187 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c26181ee-5e97-4934-a7da-71a24e303856-cluster-monitoring-operator-tls podName:c26181ee-5e97-4934-a7da-71a24e303856 nodeName:}" failed. No retries permitted until 2026-09-18 06:26:27.638154003 +0000 UTC m=+249.377570160 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c26181ee-5e97-4934-a7da-71a24e303856-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-tqkh6" (UID: "c26181ee-5e97-4934-a7da-71a24e303856") : secret "cluster-monitoring-operator-tls" not found Sep 18 06:26:27.138645 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.138218 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b-trusted-ca\") pod \"image-registry-7cfd7d874-4m95j\" (UID: \"ca057b3a-aa6d-45ff-bad8-6dc74bf2009b\") " pod="openshift-image-registry/image-registry-7cfd7d874-4m95j" Sep 18 06:26:27.138645 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.138270 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b-registry-certificates\") pod \"image-registry-7cfd7d874-4m95j\" (UID: \"ca057b3a-aa6d-45ff-bad8-6dc74bf2009b\") " pod="openshift-image-registry/image-registry-7cfd7d874-4m95j" Sep 18 06:26:27.138645 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.138301 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b856fd0d-7ea9-4899-958a-98efec61fe44-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-kzkk8\" (UID: \"b856fd0d-7ea9-4899-958a-98efec61fe44\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-kzkk8" Sep 18 06:26:27.138645 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.138342 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee4fa404-921d-48d8-a26d-fa5c15425928-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-5bn97\" (UID: \"ee4fa404-921d-48d8-a26d-fa5c15425928\") " pod="openshift-insights/insights-operator-6f9c644dd9-5bn97" Sep 18 06:26:27.138645 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.138407 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j54rs\" (UniqueName: \"kubernetes.io/projected/dcc10422-179e-4253-95a4-b8621790f350-kube-api-access-j54rs\") pod \"cluster-samples-operator-685d7d68b5-qcsfv\" (UID: \"dcc10422-179e-4253-95a4-b8621790f350\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-qcsfv" Sep 18 06:26:27.138645 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.138433 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b-bound-sa-token\") pod \"image-registry-7cfd7d874-4m95j\" (UID: \"ca057b3a-aa6d-45ff-bad8-6dc74bf2009b\") " pod="openshift-image-registry/image-registry-7cfd7d874-4m95j" Sep 18 06:26:27.138645 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.138343 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee4fa404-921d-48d8-a26d-fa5c15425928-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-5bn97\" (UID: \"ee4fa404-921d-48d8-a26d-fa5c15425928\") " pod="openshift-insights/insights-operator-6f9c644dd9-5bn97" Sep 18 06:26:27.139090 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.138810 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zngf6\" (UniqueName: \"kubernetes.io/projected/6164b435-ee87-4137-b551-f951321399e6-kube-api-access-zngf6\") pod \"volume-data-source-validator-5c4db8bdf9-fthpb\" (UID: \"6164b435-ee87-4137-b551-f951321399e6\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-fthpb" Sep 18 06:26:27.139090 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.138904 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee4fa404-921d-48d8-a26d-fa5c15425928-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-5bn97\" (UID: \"ee4fa404-921d-48d8-a26d-fa5c15425928\") " pod="openshift-insights/insights-operator-6f9c644dd9-5bn97" Sep 18 06:26:27.139090 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.138928 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c26181ee-5e97-4934-a7da-71a24e303856-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-tqkh6\" (UID: \"c26181ee-5e97-4934-a7da-71a24e303856\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-tqkh6" Sep 18 06:26:27.139090 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.138948 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b856fd0d-7ea9-4899-958a-98efec61fe44-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-kzkk8\" (UID: \"b856fd0d-7ea9-4899-958a-98efec61fe44\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-kzkk8" Sep 18 06:26:27.139090 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.138845 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ee4fa404-921d-48d8-a26d-fa5c15425928-snapshots\") pod \"insights-operator-6f9c644dd9-5bn97\" (UID: \"ee4fa404-921d-48d8-a26d-fa5c15425928\") " pod="openshift-insights/insights-operator-6f9c644dd9-5bn97" Sep 18 06:26:27.139293 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.139092 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csm7x\" (UniqueName: \"kubernetes.io/projected/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b-kube-api-access-csm7x\") pod \"image-registry-7cfd7d874-4m95j\" (UID: \"ca057b3a-aa6d-45ff-bad8-6dc74bf2009b\") " pod="openshift-image-registry/image-registry-7cfd7d874-4m95j" Sep 18 06:26:27.139741 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.139719 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ee4fa404-921d-48d8-a26d-fa5c15425928-snapshots\") pod \"insights-operator-6f9c644dd9-5bn97\" (UID: \"ee4fa404-921d-48d8-a26d-fa5c15425928\") " pod="openshift-insights/insights-operator-6f9c644dd9-5bn97" Sep 18 06:26:27.140242 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.140215 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b856fd0d-7ea9-4899-958a-98efec61fe44-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-kzkk8\" (UID: \"b856fd0d-7ea9-4899-958a-98efec61fe44\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-kzkk8" Sep 18 06:26:27.140326 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.140218 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e16d183f-ea45-4a4c-94be-f078321c3c79-default-certificate\") pod \"router-default-75b6d7b44b-fvj8r\" (UID: \"e16d183f-ea45-4a4c-94be-f078321c3c79\") " pod="openshift-ingress/router-default-75b6d7b44b-fvj8r" Sep 18 06:26:27.140326 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.140252 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ee4fa404-921d-48d8-a26d-fa5c15425928-serving-cert\") pod \"insights-operator-6f9c644dd9-5bn97\" (UID: \"ee4fa404-921d-48d8-a26d-fa5c15425928\") " pod="openshift-insights/insights-operator-6f9c644dd9-5bn97" Sep 18 06:26:27.140881 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.140863 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e16d183f-ea45-4a4c-94be-f078321c3c79-stats-auth\") pod \"router-default-75b6d7b44b-fvj8r\" (UID: \"e16d183f-ea45-4a4c-94be-f078321c3c79\") " pod="openshift-ingress/router-default-75b6d7b44b-fvj8r" Sep 18 06:26:27.148994 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.148960 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-np696\" (UniqueName: \"kubernetes.io/projected/ee4fa404-921d-48d8-a26d-fa5c15425928-kube-api-access-np696\") pod \"insights-operator-6f9c644dd9-5bn97\" (UID: \"ee4fa404-921d-48d8-a26d-fa5c15425928\") " pod="openshift-insights/insights-operator-6f9c644dd9-5bn97" Sep 18 06:26:27.149169 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.149102 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzdhh\" (UniqueName: \"kubernetes.io/projected/e16d183f-ea45-4a4c-94be-f078321c3c79-kube-api-access-dzdhh\") pod \"router-default-75b6d7b44b-fvj8r\" (UID: \"e16d183f-ea45-4a4c-94be-f078321c3c79\") " pod="openshift-ingress/router-default-75b6d7b44b-fvj8r" Sep 18 06:26:27.149169 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.149102 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j54rs\" (UniqueName: \"kubernetes.io/projected/dcc10422-179e-4253-95a4-b8621790f350-kube-api-access-j54rs\") pod \"cluster-samples-operator-685d7d68b5-qcsfv\" (UID: \"dcc10422-179e-4253-95a4-b8621790f350\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-qcsfv" Sep 18 06:26:27.149682 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.149605 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg25l\" (UniqueName: \"kubernetes.io/projected/b856fd0d-7ea9-4899-958a-98efec61fe44-kube-api-access-cg25l\") pod \"kube-storage-version-migrator-operator-55889f7b47-kzkk8\" (UID: \"b856fd0d-7ea9-4899-958a-98efec61fe44\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-kzkk8" Sep 18 06:26:27.150099 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.150079 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfprn\" (UniqueName: \"kubernetes.io/projected/c26181ee-5e97-4934-a7da-71a24e303856-kube-api-access-pfprn\") pod \"cluster-monitoring-operator-847b8bd8d7-tqkh6\" (UID: \"c26181ee-5e97-4934-a7da-71a24e303856\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-tqkh6" Sep 18 06:26:27.150171 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.150111 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zngf6\" (UniqueName: \"kubernetes.io/projected/6164b435-ee87-4137-b551-f951321399e6-kube-api-access-zngf6\") pod \"volume-data-source-validator-5c4db8bdf9-fthpb\" (UID: \"6164b435-ee87-4137-b551-f951321399e6\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-fthpb" Sep 18 06:26:27.238720 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.238634 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-5bn97" Sep 18 06:26:27.239869 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.239847 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b-registry-tls\") pod \"image-registry-7cfd7d874-4m95j\" (UID: \"ca057b3a-aa6d-45ff-bad8-6dc74bf2009b\") " pod="openshift-image-registry/image-registry-7cfd7d874-4m95j" Sep 18 06:26:27.239955 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.239877 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b-installation-pull-secrets\") pod \"image-registry-7cfd7d874-4m95j\" (UID: \"ca057b3a-aa6d-45ff-bad8-6dc74bf2009b\") " pod="openshift-image-registry/image-registry-7cfd7d874-4m95j" Sep 18 06:26:27.239955 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.239893 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b-image-registry-private-configuration\") pod \"image-registry-7cfd7d874-4m95j\" (UID: \"ca057b3a-aa6d-45ff-bad8-6dc74bf2009b\") " pod="openshift-image-registry/image-registry-7cfd7d874-4m95j" Sep 18 06:26:27.239955 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.239911 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b-ca-trust-extracted\") pod \"image-registry-7cfd7d874-4m95j\" (UID: \"ca057b3a-aa6d-45ff-bad8-6dc74bf2009b\") " pod="openshift-image-registry/image-registry-7cfd7d874-4m95j" Sep 18 06:26:27.240053 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:27.239991 2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 18 06:26:27.240053 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:27.240014 2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cfd7d874-4m95j: secret "image-registry-tls" not found Sep 18 06:26:27.240113 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.240071 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b-trusted-ca\") pod \"image-registry-7cfd7d874-4m95j\" (UID: \"ca057b3a-aa6d-45ff-bad8-6dc74bf2009b\") " pod="openshift-image-registry/image-registry-7cfd7d874-4m95j" Sep 18 06:26:27.240159 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.240118 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b-registry-certificates\") pod \"image-registry-7cfd7d874-4m95j\" (UID: \"ca057b3a-aa6d-45ff-bad8-6dc74bf2009b\") " pod="openshift-image-registry/image-registry-7cfd7d874-4m95j" Sep 18 06:26:27.240290 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:27.240259 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b-registry-tls podName:ca057b3a-aa6d-45ff-bad8-6dc74bf2009b nodeName:}" failed. No retries permitted until 2026-09-18 06:26:27.740232643 +0000 UTC m=+249.479648799 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b-registry-tls") pod "image-registry-7cfd7d874-4m95j" (UID: "ca057b3a-aa6d-45ff-bad8-6dc74bf2009b") : secret "image-registry-tls" not found Sep 18 06:26:27.240290 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.240262 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b-ca-trust-extracted\") pod \"image-registry-7cfd7d874-4m95j\" (UID: \"ca057b3a-aa6d-45ff-bad8-6dc74bf2009b\") " pod="openshift-image-registry/image-registry-7cfd7d874-4m95j" Sep 18 06:26:27.240490 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.240302 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b-bound-sa-token\") pod \"image-registry-7cfd7d874-4m95j\" (UID: \"ca057b3a-aa6d-45ff-bad8-6dc74bf2009b\") " pod="openshift-image-registry/image-registry-7cfd7d874-4m95j" Sep 18 06:26:27.240490 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.240356 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csm7x\" (UniqueName: \"kubernetes.io/projected/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b-kube-api-access-csm7x\") pod \"image-registry-7cfd7d874-4m95j\" (UID: \"ca057b3a-aa6d-45ff-bad8-6dc74bf2009b\") " pod="openshift-image-registry/image-registry-7cfd7d874-4m95j" Sep 18 06:26:27.240763 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.240729 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b-registry-certificates\") pod \"image-registry-7cfd7d874-4m95j\" (UID: \"ca057b3a-aa6d-45ff-bad8-6dc74bf2009b\") " pod="openshift-image-registry/image-registry-7cfd7d874-4m95j" Sep 18 06:26:27.241123 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.241101 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b-trusted-ca\") pod \"image-registry-7cfd7d874-4m95j\" (UID: \"ca057b3a-aa6d-45ff-bad8-6dc74bf2009b\") " pod="openshift-image-registry/image-registry-7cfd7d874-4m95j" Sep 18 06:26:27.242576 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.242555 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b-image-registry-private-configuration\") pod \"image-registry-7cfd7d874-4m95j\" (UID: \"ca057b3a-aa6d-45ff-bad8-6dc74bf2009b\") " pod="openshift-image-registry/image-registry-7cfd7d874-4m95j" Sep 18 06:26:27.242671 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.242655 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b-installation-pull-secrets\") pod \"image-registry-7cfd7d874-4m95j\" (UID: \"ca057b3a-aa6d-45ff-bad8-6dc74bf2009b\") " pod="openshift-image-registry/image-registry-7cfd7d874-4m95j" Sep 18 06:26:27.248670 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.248651 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b-bound-sa-token\") pod \"image-registry-7cfd7d874-4m95j\" (UID: \"ca057b3a-aa6d-45ff-bad8-6dc74bf2009b\") " pod="openshift-image-registry/image-registry-7cfd7d874-4m95j" Sep 18 06:26:27.249093 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.249073 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csm7x\" (UniqueName: \"kubernetes.io/projected/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b-kube-api-access-csm7x\") pod \"image-registry-7cfd7d874-4m95j\" (UID: \"ca057b3a-aa6d-45ff-bad8-6dc74bf2009b\") " pod="openshift-image-registry/image-registry-7cfd7d874-4m95j" Sep 18 06:26:27.312910 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.312879 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-kzkk8" Sep 18 06:26:27.319729 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.319657 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-fthpb" Sep 18 06:26:27.508396 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.508345 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-5bn97"] Sep 18 06:26:27.510834 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:26:27.510806 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee4fa404_921d_48d8_a26d_fa5c15425928.slice/crio-0fa71433a47e7a6aac5f49cd4d1535141af768bb3ebcb243aaeb196898f5d054 WatchSource:0}: Error finding container 0fa71433a47e7a6aac5f49cd4d1535141af768bb3ebcb243aaeb196898f5d054: Status 404 returned error can't find the container with id 0fa71433a47e7a6aac5f49cd4d1535141af768bb3ebcb243aaeb196898f5d054 Sep 18 06:26:27.581196 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.581151 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-kzkk8"] Sep 18 06:26:27.584449 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:26:27.584415 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb856fd0d_7ea9_4899_958a_98efec61fe44.slice/crio-74b36b502717f0c0c22999ebb0f3591afb214ed3103c22d30054028a318bc32a WatchSource:0}: Error finding container 74b36b502717f0c0c22999ebb0f3591afb214ed3103c22d30054028a318bc32a: Status 404 returned error can't find the container with id 74b36b502717f0c0c22999ebb0f3591afb214ed3103c22d30054028a318bc32a Sep 18 06:26:27.600195 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.600171 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-fthpb"] Sep 18 06:26:27.603427 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:26:27.603403 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6164b435_ee87_4137_b551_f951321399e6.slice/crio-5db490e8494c269c3632c1ea343932550c26125513bdf3ae3e64bcc4b92fabe4 WatchSource:0}: Error finding container 5db490e8494c269c3632c1ea343932550c26125513bdf3ae3e64bcc4b92fabe4: Status 404 returned error can't find the container with id 5db490e8494c269c3632c1ea343932550c26125513bdf3ae3e64bcc4b92fabe4 Sep 18 06:26:27.643021 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.642989 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e16d183f-ea45-4a4c-94be-f078321c3c79-metrics-certs\") pod \"router-default-75b6d7b44b-fvj8r\" (UID: \"e16d183f-ea45-4a4c-94be-f078321c3c79\") " pod="openshift-ingress/router-default-75b6d7b44b-fvj8r" Sep 18 06:26:27.643183 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.643056 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e16d183f-ea45-4a4c-94be-f078321c3c79-service-ca-bundle\") pod \"router-default-75b6d7b44b-fvj8r\" (UID: \"e16d183f-ea45-4a4c-94be-f078321c3c79\") " pod="openshift-ingress/router-default-75b6d7b44b-fvj8r" Sep 18 06:26:27.643183 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.643106 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c26181ee-5e97-4934-a7da-71a24e303856-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-tqkh6\" (UID: \"c26181ee-5e97-4934-a7da-71a24e303856\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-tqkh6" Sep 18 06:26:27.643183 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:27.643141 2567 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 18 06:26:27.643183 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:27.643169 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e16d183f-ea45-4a4c-94be-f078321c3c79-service-ca-bundle podName:e16d183f-ea45-4a4c-94be-f078321c3c79 nodeName:}" failed. No retries permitted until 2026-09-18 06:26:28.643152634 +0000 UTC m=+250.382568791 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e16d183f-ea45-4a4c-94be-f078321c3c79-service-ca-bundle") pod "router-default-75b6d7b44b-fvj8r" (UID: "e16d183f-ea45-4a4c-94be-f078321c3c79") : configmap references non-existent config key: service-ca.crt Sep 18 06:26:27.643345 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:27.643197 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e16d183f-ea45-4a4c-94be-f078321c3c79-metrics-certs podName:e16d183f-ea45-4a4c-94be-f078321c3c79 nodeName:}" failed. No retries permitted until 2026-09-18 06:26:28.64319044 +0000 UTC m=+250.382606592 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e16d183f-ea45-4a4c-94be-f078321c3c79-metrics-certs") pod "router-default-75b6d7b44b-fvj8r" (UID: "e16d183f-ea45-4a4c-94be-f078321c3c79") : secret "router-metrics-certs-default" not found Sep 18 06:26:27.643345 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:27.643205 2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 18 06:26:27.643345 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.643225 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dcc10422-179e-4253-95a4-b8621790f350-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-qcsfv\" (UID: \"dcc10422-179e-4253-95a4-b8621790f350\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-qcsfv" Sep 18 06:26:27.643345 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:27.643248 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c26181ee-5e97-4934-a7da-71a24e303856-cluster-monitoring-operator-tls podName:c26181ee-5e97-4934-a7da-71a24e303856 nodeName:}" failed. No retries permitted until 2026-09-18 06:26:28.643231734 +0000 UTC m=+250.382647900 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c26181ee-5e97-4934-a7da-71a24e303856-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-tqkh6" (UID: "c26181ee-5e97-4934-a7da-71a24e303856") : secret "cluster-monitoring-operator-tls" not found Sep 18 06:26:27.643345 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:27.643273 2567 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 18 06:26:27.643345 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:27.643296 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcc10422-179e-4253-95a4-b8621790f350-samples-operator-tls podName:dcc10422-179e-4253-95a4-b8621790f350 nodeName:}" failed. No retries permitted until 2026-09-18 06:26:28.643289284 +0000 UTC m=+250.382705436 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/dcc10422-179e-4253-95a4-b8621790f350-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-qcsfv" (UID: "dcc10422-179e-4253-95a4-b8621790f350") : secret "samples-operator-tls" not found Sep 18 06:26:27.743715 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:27.743676 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b-registry-tls\") pod \"image-registry-7cfd7d874-4m95j\" (UID: \"ca057b3a-aa6d-45ff-bad8-6dc74bf2009b\") " pod="openshift-image-registry/image-registry-7cfd7d874-4m95j" Sep 18 06:26:27.743875 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:27.743832 2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 18 06:26:27.743875 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:27.743856 2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cfd7d874-4m95j: secret "image-registry-tls" not found Sep 18 06:26:27.743941 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:27.743912 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b-registry-tls podName:ca057b3a-aa6d-45ff-bad8-6dc74bf2009b nodeName:}" failed. No retries permitted until 2026-09-18 06:26:28.743895675 +0000 UTC m=+250.483311850 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b-registry-tls") pod "image-registry-7cfd7d874-4m95j" (UID: "ca057b3a-aa6d-45ff-bad8-6dc74bf2009b") : secret "image-registry-tls" not found Sep 18 06:26:28.154928 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:28.154894 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-fthpb" event={"ID":"6164b435-ee87-4137-b551-f951321399e6","Type":"ContainerStarted","Data":"5db490e8494c269c3632c1ea343932550c26125513bdf3ae3e64bcc4b92fabe4"} Sep 18 06:26:28.156185 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:28.156154 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-kzkk8" event={"ID":"b856fd0d-7ea9-4899-958a-98efec61fe44","Type":"ContainerStarted","Data":"74b36b502717f0c0c22999ebb0f3591afb214ed3103c22d30054028a318bc32a"} Sep 18 06:26:28.157349 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:28.157324 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-5bn97" event={"ID":"ee4fa404-921d-48d8-a26d-fa5c15425928","Type":"ContainerStarted","Data":"0fa71433a47e7a6aac5f49cd4d1535141af768bb3ebcb243aaeb196898f5d054"} Sep 18 06:26:28.650747 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:28.650641 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e16d183f-ea45-4a4c-94be-f078321c3c79-service-ca-bundle\") pod \"router-default-75b6d7b44b-fvj8r\" (UID: \"e16d183f-ea45-4a4c-94be-f078321c3c79\") " pod="openshift-ingress/router-default-75b6d7b44b-fvj8r" Sep 18 06:26:28.650747 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:28.650719 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c26181ee-5e97-4934-a7da-71a24e303856-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-tqkh6\" (UID: \"c26181ee-5e97-4934-a7da-71a24e303856\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-tqkh6" Sep 18 06:26:28.650997 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:28.650791 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e16d183f-ea45-4a4c-94be-f078321c3c79-service-ca-bundle podName:e16d183f-ea45-4a4c-94be-f078321c3c79 nodeName:}" failed. No retries permitted until 2026-09-18 06:26:30.650771295 +0000 UTC m=+252.390187460 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e16d183f-ea45-4a4c-94be-f078321c3c79-service-ca-bundle") pod "router-default-75b6d7b44b-fvj8r" (UID: "e16d183f-ea45-4a4c-94be-f078321c3c79") : configmap references non-existent config key: service-ca.crt Sep 18 06:26:28.650997 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:28.650842 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dcc10422-179e-4253-95a4-b8621790f350-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-qcsfv\" (UID: \"dcc10422-179e-4253-95a4-b8621790f350\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-qcsfv" Sep 18 06:26:28.650997 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:28.650873 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e16d183f-ea45-4a4c-94be-f078321c3c79-metrics-certs\") pod \"router-default-75b6d7b44b-fvj8r\" (UID: \"e16d183f-ea45-4a4c-94be-f078321c3c79\") " pod="openshift-ingress/router-default-75b6d7b44b-fvj8r" Sep 18 06:26:28.651158 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:28.650848 2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 18 06:26:28.651158 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:28.651069 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c26181ee-5e97-4934-a7da-71a24e303856-cluster-monitoring-operator-tls podName:c26181ee-5e97-4934-a7da-71a24e303856 nodeName:}" failed. No retries permitted until 2026-09-18 06:26:30.65105694 +0000 UTC m=+252.390473092 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c26181ee-5e97-4934-a7da-71a24e303856-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-tqkh6" (UID: "c26181ee-5e97-4934-a7da-71a24e303856") : secret "cluster-monitoring-operator-tls" not found Sep 18 06:26:28.651158 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:28.650982 2567 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 18 06:26:28.651158 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:28.651100 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e16d183f-ea45-4a4c-94be-f078321c3c79-metrics-certs podName:e16d183f-ea45-4a4c-94be-f078321c3c79 nodeName:}" failed. No retries permitted until 2026-09-18 06:26:30.65109174 +0000 UTC m=+252.390507906 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e16d183f-ea45-4a4c-94be-f078321c3c79-metrics-certs") pod "router-default-75b6d7b44b-fvj8r" (UID: "e16d183f-ea45-4a4c-94be-f078321c3c79") : secret "router-metrics-certs-default" not found Sep 18 06:26:28.651158 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:28.651027 2567 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 18 06:26:28.651158 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:28.651126 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcc10422-179e-4253-95a4-b8621790f350-samples-operator-tls podName:dcc10422-179e-4253-95a4-b8621790f350 nodeName:}" failed. No retries permitted until 2026-09-18 06:26:30.651118604 +0000 UTC m=+252.390534756 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/dcc10422-179e-4253-95a4-b8621790f350-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-qcsfv" (UID: "dcc10422-179e-4253-95a4-b8621790f350") : secret "samples-operator-tls" not found Sep 18 06:26:28.752000 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:28.751930 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b-registry-tls\") pod \"image-registry-7cfd7d874-4m95j\" (UID: \"ca057b3a-aa6d-45ff-bad8-6dc74bf2009b\") " pod="openshift-image-registry/image-registry-7cfd7d874-4m95j" Sep 18 06:26:28.752182 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:28.752055 2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 18 06:26:28.752182 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:28.752081 2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cfd7d874-4m95j: secret "image-registry-tls" not found Sep 18 06:26:28.752182 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:28.752148 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b-registry-tls podName:ca057b3a-aa6d-45ff-bad8-6dc74bf2009b nodeName:}" failed. No retries permitted until 2026-09-18 06:26:30.75212788 +0000 UTC m=+252.491544054 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b-registry-tls") pod "image-registry-7cfd7d874-4m95j" (UID: "ca057b3a-aa6d-45ff-bad8-6dc74bf2009b") : secret "image-registry-tls" not found Sep 18 06:26:29.163106 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:29.162988 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-fthpb" event={"ID":"6164b435-ee87-4137-b551-f951321399e6","Type":"ContainerStarted","Data":"fd42aa9efa8418841f89f6ed49ef2b82e3c9df01b38227df4e878fc60ea4fe05"} Sep 18 06:26:29.178683 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:29.178609 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-fthpb" podStartSLOduration=1.8882332530000001 podStartE2EDuration="3.178594697s" totalImagesPullingTime="1.290361444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:26:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:26:29.176887958 +0000 UTC m=+250.916304151" watchObservedRunningTime="2026-09-18 06:26:29.178594697 +0000 UTC m=+250.918010865" Sep 18 06:26:30.169118 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:30.169082 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-kzkk8" event={"ID":"b856fd0d-7ea9-4899-958a-98efec61fe44","Type":"ContainerStarted","Data":"e69cfdab794fe6fb2cf15abeaf2753b437a5e214c10fedf8ea96fb9fb3d53783"} Sep 18 06:26:30.170922 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:30.170893 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-5bn97" event={"ID":"ee4fa404-921d-48d8-a26d-fa5c15425928","Type":"ContainerStarted","Data":"d3014bd3c84bbf53ce29dea05cdbd75bae13df6a5461088e2923113348873169"} Sep 18 06:26:30.183686 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:30.183640 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-kzkk8" podStartSLOduration=1.706087087 podStartE2EDuration="4.18362402s" totalImagesPullingTime="2.477536933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:26:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:26:30.182707311 +0000 UTC m=+251.922123512" watchObservedRunningTime="2026-09-18 06:26:30.18362402 +0000 UTC m=+251.923040194" Sep 18 06:26:30.200154 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:30.200099 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-6f9c644dd9-5bn97" podStartSLOduration=1.647481343 podStartE2EDuration="4.200079221s" totalImagesPullingTime="2.552597878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:26:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:26:30.199103998 +0000 UTC m=+251.938520178" watchObservedRunningTime="2026-09-18 06:26:30.200079221 +0000 UTC m=+251.939495402" Sep 18 06:26:30.667727 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:30.667689 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dcc10422-179e-4253-95a4-b8621790f350-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-qcsfv\" (UID: \"dcc10422-179e-4253-95a4-b8621790f350\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-qcsfv" Sep 18 06:26:30.667727 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:30.667726 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e16d183f-ea45-4a4c-94be-f078321c3c79-metrics-certs\") pod \"router-default-75b6d7b44b-fvj8r\" (UID: \"e16d183f-ea45-4a4c-94be-f078321c3c79\") " pod="openshift-ingress/router-default-75b6d7b44b-fvj8r" Sep 18 06:26:30.667961 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:30.667788 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e16d183f-ea45-4a4c-94be-f078321c3c79-service-ca-bundle\") pod \"router-default-75b6d7b44b-fvj8r\" (UID: \"e16d183f-ea45-4a4c-94be-f078321c3c79\") " pod="openshift-ingress/router-default-75b6d7b44b-fvj8r" Sep 18 06:26:30.667961 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:30.667814 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c26181ee-5e97-4934-a7da-71a24e303856-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-tqkh6\" (UID: \"c26181ee-5e97-4934-a7da-71a24e303856\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-tqkh6" Sep 18 06:26:30.667961 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:30.667854 2567 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 18 06:26:30.667961 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:30.667905 2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 18 06:26:30.667961 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:30.667921 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcc10422-179e-4253-95a4-b8621790f350-samples-operator-tls podName:dcc10422-179e-4253-95a4-b8621790f350 nodeName:}" failed. No retries permitted until 2026-09-18 06:26:34.667901179 +0000 UTC m=+256.407317336 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/dcc10422-179e-4253-95a4-b8621790f350-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-qcsfv" (UID: "dcc10422-179e-4253-95a4-b8621790f350") : secret "samples-operator-tls" not found Sep 18 06:26:30.667961 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:30.667938 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e16d183f-ea45-4a4c-94be-f078321c3c79-service-ca-bundle podName:e16d183f-ea45-4a4c-94be-f078321c3c79 nodeName:}" failed. No retries permitted until 2026-09-18 06:26:34.667928984 +0000 UTC m=+256.407345136 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e16d183f-ea45-4a4c-94be-f078321c3c79-service-ca-bundle") pod "router-default-75b6d7b44b-fvj8r" (UID: "e16d183f-ea45-4a4c-94be-f078321c3c79") : configmap references non-existent config key: service-ca.crt Sep 18 06:26:30.667961 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:30.667949 2567 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 18 06:26:30.668334 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:30.667953 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c26181ee-5e97-4934-a7da-71a24e303856-cluster-monitoring-operator-tls podName:c26181ee-5e97-4934-a7da-71a24e303856 nodeName:}" failed. No retries permitted until 2026-09-18 06:26:34.66794507 +0000 UTC m=+256.407361228 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c26181ee-5e97-4934-a7da-71a24e303856-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-tqkh6" (UID: "c26181ee-5e97-4934-a7da-71a24e303856") : secret "cluster-monitoring-operator-tls" not found Sep 18 06:26:30.668334 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:30.667999 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e16d183f-ea45-4a4c-94be-f078321c3c79-metrics-certs podName:e16d183f-ea45-4a4c-94be-f078321c3c79 nodeName:}" failed. No retries permitted until 2026-09-18 06:26:34.667984869 +0000 UTC m=+256.407401037 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e16d183f-ea45-4a4c-94be-f078321c3c79-metrics-certs") pod "router-default-75b6d7b44b-fvj8r" (UID: "e16d183f-ea45-4a4c-94be-f078321c3c79") : secret "router-metrics-certs-default" not found Sep 18 06:26:30.768570 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:30.768534 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b-registry-tls\") pod \"image-registry-7cfd7d874-4m95j\" (UID: \"ca057b3a-aa6d-45ff-bad8-6dc74bf2009b\") " pod="openshift-image-registry/image-registry-7cfd7d874-4m95j" Sep 18 06:26:30.768731 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:30.768689 2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 18 06:26:30.768731 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:30.768711 2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cfd7d874-4m95j: secret "image-registry-tls" not found Sep 18 06:26:30.768851 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:30.768771 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b-registry-tls podName:ca057b3a-aa6d-45ff-bad8-6dc74bf2009b nodeName:}" failed. No retries permitted until 2026-09-18 06:26:34.768750987 +0000 UTC m=+256.508167141 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b-registry-tls") pod "image-registry-7cfd7d874-4m95j" (UID: "ca057b3a-aa6d-45ff-bad8-6dc74bf2009b") : secret "image-registry-tls" not found Sep 18 06:26:31.954421 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:31.954385 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6nf5m"] Sep 18 06:26:31.957061 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:31.957005 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6nf5m" Sep 18 06:26:31.959358 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:31.959332 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6n6ts\"" Sep 18 06:26:31.959749 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:31.959726 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 18 06:26:31.960292 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:31.960274 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 18 06:26:31.968446 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:31.968425 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6nf5m"] Sep 18 06:26:32.080584 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:32.080555 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b5029a64-fc57-44f7-b2e8-79c9a82c572d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6nf5m\" (UID: \"b5029a64-fc57-44f7-b2e8-79c9a82c572d\") " pod="openshift-insights/insights-runtime-extractor-6nf5m" Sep 18 06:26:32.080765 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:32.080625 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b5029a64-fc57-44f7-b2e8-79c9a82c572d-data-volume\") pod \"insights-runtime-extractor-6nf5m\" (UID: \"b5029a64-fc57-44f7-b2e8-79c9a82c572d\") " pod="openshift-insights/insights-runtime-extractor-6nf5m" Sep 18 06:26:32.080765 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:32.080647 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b5029a64-fc57-44f7-b2e8-79c9a82c572d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6nf5m\" (UID: \"b5029a64-fc57-44f7-b2e8-79c9a82c572d\") " pod="openshift-insights/insights-runtime-extractor-6nf5m" Sep 18 06:26:32.080765 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:32.080727 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpqrc\" (UniqueName: \"kubernetes.io/projected/b5029a64-fc57-44f7-b2e8-79c9a82c572d-kube-api-access-mpqrc\") pod \"insights-runtime-extractor-6nf5m\" (UID: \"b5029a64-fc57-44f7-b2e8-79c9a82c572d\") " pod="openshift-insights/insights-runtime-extractor-6nf5m" Sep 18 06:26:32.080908 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:32.080778 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b5029a64-fc57-44f7-b2e8-79c9a82c572d-crio-socket\") pod \"insights-runtime-extractor-6nf5m\" (UID: \"b5029a64-fc57-44f7-b2e8-79c9a82c572d\") " pod="openshift-insights/insights-runtime-extractor-6nf5m" Sep 18 06:26:32.182015 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:32.181983 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b5029a64-fc57-44f7-b2e8-79c9a82c572d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6nf5m\" (UID: \"b5029a64-fc57-44f7-b2e8-79c9a82c572d\") " pod="openshift-insights/insights-runtime-extractor-6nf5m" Sep 18 06:26:32.182176 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:32.182022 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpqrc\" (UniqueName: \"kubernetes.io/projected/b5029a64-fc57-44f7-b2e8-79c9a82c572d-kube-api-access-mpqrc\") pod \"insights-runtime-extractor-6nf5m\" (UID: \"b5029a64-fc57-44f7-b2e8-79c9a82c572d\") " pod="openshift-insights/insights-runtime-extractor-6nf5m" Sep 18 06:26:32.182176 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:32.182046 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b5029a64-fc57-44f7-b2e8-79c9a82c572d-crio-socket\") pod \"insights-runtime-extractor-6nf5m\" (UID: \"b5029a64-fc57-44f7-b2e8-79c9a82c572d\") " pod="openshift-insights/insights-runtime-extractor-6nf5m" Sep 18 06:26:32.182176 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:32.182113 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 18 06:26:32.182357 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:32.182120 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b5029a64-fc57-44f7-b2e8-79c9a82c572d-crio-socket\") pod \"insights-runtime-extractor-6nf5m\" (UID: \"b5029a64-fc57-44f7-b2e8-79c9a82c572d\") " pod="openshift-insights/insights-runtime-extractor-6nf5m" Sep 18 06:26:32.182357 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:32.182198 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b5029a64-fc57-44f7-b2e8-79c9a82c572d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6nf5m\" (UID: \"b5029a64-fc57-44f7-b2e8-79c9a82c572d\") " pod="openshift-insights/insights-runtime-extractor-6nf5m" Sep 18 06:26:32.182357 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:32.182232 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b5029a64-fc57-44f7-b2e8-79c9a82c572d-kube-rbac-proxy-cm podName:b5029a64-fc57-44f7-b2e8-79c9a82c572d nodeName:}" failed. No retries permitted until 2026-09-18 06:26:32.682211501 +0000 UTC m=+254.421627653 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b5029a64-fc57-44f7-b2e8-79c9a82c572d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6nf5m" (UID: "b5029a64-fc57-44f7-b2e8-79c9a82c572d") : configmap "kube-rbac-proxy" not found Sep 18 06:26:32.182357 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:32.182272 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 18 06:26:32.182357 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:32.182307 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b5029a64-fc57-44f7-b2e8-79c9a82c572d-data-volume\") pod \"insights-runtime-extractor-6nf5m\" (UID: \"b5029a64-fc57-44f7-b2e8-79c9a82c572d\") " pod="openshift-insights/insights-runtime-extractor-6nf5m" Sep 18 06:26:32.182357 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:32.182321 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5029a64-fc57-44f7-b2e8-79c9a82c572d-insights-runtime-extractor-tls podName:b5029a64-fc57-44f7-b2e8-79c9a82c572d nodeName:}" failed. No retries permitted until 2026-09-18 06:26:32.682308782 +0000 UTC m=+254.421724933 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b5029a64-fc57-44f7-b2e8-79c9a82c572d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6nf5m" (UID: "b5029a64-fc57-44f7-b2e8-79c9a82c572d") : secret "insights-runtime-extractor-tls" not found Sep 18 06:26:32.182601 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:32.182585 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b5029a64-fc57-44f7-b2e8-79c9a82c572d-data-volume\") pod \"insights-runtime-extractor-6nf5m\" (UID: \"b5029a64-fc57-44f7-b2e8-79c9a82c572d\") " pod="openshift-insights/insights-runtime-extractor-6nf5m" Sep 18 06:26:32.189452 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:32.189430 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpqrc\" (UniqueName: \"kubernetes.io/projected/b5029a64-fc57-44f7-b2e8-79c9a82c572d-kube-api-access-mpqrc\") pod \"insights-runtime-extractor-6nf5m\" (UID: \"b5029a64-fc57-44f7-b2e8-79c9a82c572d\") " pod="openshift-insights/insights-runtime-extractor-6nf5m" Sep 18 06:26:32.383746 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:32.383717 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66dca255-1289-49ec-b200-e950a30e3d9c-metrics-certs\") pod \"network-metrics-daemon-k77fk\" (UID: \"66dca255-1289-49ec-b200-e950a30e3d9c\") " pod="openshift-multus/network-metrics-daemon-k77fk" Sep 18 06:26:32.383957 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:32.383848 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 18 06:26:32.383957 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:32.383897 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66dca255-1289-49ec-b200-e950a30e3d9c-metrics-certs podName:66dca255-1289-49ec-b200-e950a30e3d9c nodeName:}" failed. No retries permitted until 2026-09-18 06:28:34.383884966 +0000 UTC m=+376.123301122 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66dca255-1289-49ec-b200-e950a30e3d9c-metrics-certs") pod "network-metrics-daemon-k77fk" (UID: "66dca255-1289-49ec-b200-e950a30e3d9c") : secret "metrics-daemon-secret" not found Sep 18 06:26:32.685736 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:32.685635 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b5029a64-fc57-44f7-b2e8-79c9a82c572d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6nf5m\" (UID: \"b5029a64-fc57-44f7-b2e8-79c9a82c572d\") " pod="openshift-insights/insights-runtime-extractor-6nf5m" Sep 18 06:26:32.685906 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:32.685747 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b5029a64-fc57-44f7-b2e8-79c9a82c572d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6nf5m\" (UID: \"b5029a64-fc57-44f7-b2e8-79c9a82c572d\") " pod="openshift-insights/insights-runtime-extractor-6nf5m" Sep 18 06:26:32.685906 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:32.685802 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 18 06:26:32.685906 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:32.685833 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 18 06:26:32.685906 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:32.685888 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b5029a64-fc57-44f7-b2e8-79c9a82c572d-kube-rbac-proxy-cm podName:b5029a64-fc57-44f7-b2e8-79c9a82c572d nodeName:}" failed. No retries permitted until 2026-09-18 06:26:33.685868638 +0000 UTC m=+255.425284793 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b5029a64-fc57-44f7-b2e8-79c9a82c572d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6nf5m" (UID: "b5029a64-fc57-44f7-b2e8-79c9a82c572d") : configmap "kube-rbac-proxy" not found Sep 18 06:26:32.686073 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:32.685918 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5029a64-fc57-44f7-b2e8-79c9a82c572d-insights-runtime-extractor-tls podName:b5029a64-fc57-44f7-b2e8-79c9a82c572d nodeName:}" failed. No retries permitted until 2026-09-18 06:26:33.685899488 +0000 UTC m=+255.425315655 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b5029a64-fc57-44f7-b2e8-79c9a82c572d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6nf5m" (UID: "b5029a64-fc57-44f7-b2e8-79c9a82c572d") : secret "insights-runtime-extractor-tls" not found Sep 18 06:26:33.692432 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:33.692390 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b5029a64-fc57-44f7-b2e8-79c9a82c572d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6nf5m\" (UID: \"b5029a64-fc57-44f7-b2e8-79c9a82c572d\") " pod="openshift-insights/insights-runtime-extractor-6nf5m" Sep 18 06:26:33.692866 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:33.692499 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b5029a64-fc57-44f7-b2e8-79c9a82c572d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6nf5m\" (UID: \"b5029a64-fc57-44f7-b2e8-79c9a82c572d\") " pod="openshift-insights/insights-runtime-extractor-6nf5m" Sep 18 06:26:33.692866 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:33.692543 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 18 06:26:33.692866 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:33.692615 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5029a64-fc57-44f7-b2e8-79c9a82c572d-insights-runtime-extractor-tls podName:b5029a64-fc57-44f7-b2e8-79c9a82c572d nodeName:}" failed. No retries permitted until 2026-09-18 06:26:35.692599004 +0000 UTC m=+257.432015161 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b5029a64-fc57-44f7-b2e8-79c9a82c572d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6nf5m" (UID: "b5029a64-fc57-44f7-b2e8-79c9a82c572d") : secret "insights-runtime-extractor-tls" not found Sep 18 06:26:33.692866 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:33.692614 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 18 06:26:33.692866 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:33.692667 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b5029a64-fc57-44f7-b2e8-79c9a82c572d-kube-rbac-proxy-cm podName:b5029a64-fc57-44f7-b2e8-79c9a82c572d nodeName:}" failed. No retries permitted until 2026-09-18 06:26:35.692651674 +0000 UTC m=+257.432067840 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b5029a64-fc57-44f7-b2e8-79c9a82c572d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6nf5m" (UID: "b5029a64-fc57-44f7-b2e8-79c9a82c572d") : configmap "kube-rbac-proxy" not found Sep 18 06:26:34.521596 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:34.521569 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kgx9w_c7e27a94-0a48-466c-abdb-9f6b0758e358/dns-node-resolver/0.log" Sep 18 06:26:34.701051 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:34.701017 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dcc10422-179e-4253-95a4-b8621790f350-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-qcsfv\" (UID: \"dcc10422-179e-4253-95a4-b8621790f350\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-qcsfv" Sep 18 06:26:34.701051 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:34.701053 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e16d183f-ea45-4a4c-94be-f078321c3c79-metrics-certs\") pod \"router-default-75b6d7b44b-fvj8r\" (UID: \"e16d183f-ea45-4a4c-94be-f078321c3c79\") " pod="openshift-ingress/router-default-75b6d7b44b-fvj8r" Sep 18 06:26:34.701528 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:34.701103 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e16d183f-ea45-4a4c-94be-f078321c3c79-service-ca-bundle\") pod \"router-default-75b6d7b44b-fvj8r\" (UID: \"e16d183f-ea45-4a4c-94be-f078321c3c79\") " pod="openshift-ingress/router-default-75b6d7b44b-fvj8r" Sep 18 06:26:34.701528 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:34.701131 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c26181ee-5e97-4934-a7da-71a24e303856-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-tqkh6\" (UID: \"c26181ee-5e97-4934-a7da-71a24e303856\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-tqkh6" Sep 18 06:26:34.701528 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:34.701173 2567 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 18 06:26:34.701528 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:34.701208 2567 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 18 06:26:34.701528 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:34.701229 2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 18 06:26:34.701528 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:34.701243 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcc10422-179e-4253-95a4-b8621790f350-samples-operator-tls podName:dcc10422-179e-4253-95a4-b8621790f350 nodeName:}" failed. No retries permitted until 2026-09-18 06:26:42.701226185 +0000 UTC m=+264.440642338 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/dcc10422-179e-4253-95a4-b8621790f350-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-qcsfv" (UID: "dcc10422-179e-4253-95a4-b8621790f350") : secret "samples-operator-tls" not found Sep 18 06:26:34.701528 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:34.701261 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e16d183f-ea45-4a4c-94be-f078321c3c79-metrics-certs podName:e16d183f-ea45-4a4c-94be-f078321c3c79 nodeName:}" failed. No retries permitted until 2026-09-18 06:26:42.701250734 +0000 UTC m=+264.440666889 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e16d183f-ea45-4a4c-94be-f078321c3c79-metrics-certs") pod "router-default-75b6d7b44b-fvj8r" (UID: "e16d183f-ea45-4a4c-94be-f078321c3c79") : secret "router-metrics-certs-default" not found Sep 18 06:26:34.701528 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:34.701274 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c26181ee-5e97-4934-a7da-71a24e303856-cluster-monitoring-operator-tls podName:c26181ee-5e97-4934-a7da-71a24e303856 nodeName:}" failed. No retries permitted until 2026-09-18 06:26:42.701268686 +0000 UTC m=+264.440684838 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c26181ee-5e97-4934-a7da-71a24e303856-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-tqkh6" (UID: "c26181ee-5e97-4934-a7da-71a24e303856") : secret "cluster-monitoring-operator-tls" not found Sep 18 06:26:34.701528 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:34.701285 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e16d183f-ea45-4a4c-94be-f078321c3c79-service-ca-bundle podName:e16d183f-ea45-4a4c-94be-f078321c3c79 nodeName:}" failed. No retries permitted until 2026-09-18 06:26:42.701280096 +0000 UTC m=+264.440696248 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e16d183f-ea45-4a4c-94be-f078321c3c79-service-ca-bundle") pod "router-default-75b6d7b44b-fvj8r" (UID: "e16d183f-ea45-4a4c-94be-f078321c3c79") : configmap references non-existent config key: service-ca.crt Sep 18 06:26:34.801968 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:34.801894 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b-registry-tls\") pod \"image-registry-7cfd7d874-4m95j\" (UID: \"ca057b3a-aa6d-45ff-bad8-6dc74bf2009b\") " pod="openshift-image-registry/image-registry-7cfd7d874-4m95j" Sep 18 06:26:34.802107 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:34.802011 2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 18 06:26:34.802107 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:34.802021 2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cfd7d874-4m95j: secret "image-registry-tls" not found Sep 18 06:26:34.802107 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:34.802067 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b-registry-tls podName:ca057b3a-aa6d-45ff-bad8-6dc74bf2009b nodeName:}" failed. No retries permitted until 2026-09-18 06:26:42.802054469 +0000 UTC m=+264.541470621 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b-registry-tls") pod "image-registry-7cfd7d874-4m95j" (UID: "ca057b3a-aa6d-45ff-bad8-6dc74bf2009b") : secret "image-registry-tls" not found Sep 18 06:26:35.709113 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:35.709084 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b5029a64-fc57-44f7-b2e8-79c9a82c572d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6nf5m\" (UID: \"b5029a64-fc57-44f7-b2e8-79c9a82c572d\") " pod="openshift-insights/insights-runtime-extractor-6nf5m" Sep 18 06:26:35.709523 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:35.709177 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b5029a64-fc57-44f7-b2e8-79c9a82c572d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6nf5m\" (UID: \"b5029a64-fc57-44f7-b2e8-79c9a82c572d\") " pod="openshift-insights/insights-runtime-extractor-6nf5m" Sep 18 06:26:35.709523 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:35.709205 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 18 06:26:35.709523 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:35.709260 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b5029a64-fc57-44f7-b2e8-79c9a82c572d-kube-rbac-proxy-cm podName:b5029a64-fc57-44f7-b2e8-79c9a82c572d nodeName:}" failed. No retries permitted until 2026-09-18 06:26:39.709245874 +0000 UTC m=+261.448662036 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b5029a64-fc57-44f7-b2e8-79c9a82c572d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6nf5m" (UID: "b5029a64-fc57-44f7-b2e8-79c9a82c572d") : configmap "kube-rbac-proxy" not found Sep 18 06:26:35.709523 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:35.709322 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 18 06:26:35.709523 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:35.709399 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5029a64-fc57-44f7-b2e8-79c9a82c572d-insights-runtime-extractor-tls podName:b5029a64-fc57-44f7-b2e8-79c9a82c572d nodeName:}" failed. No retries permitted until 2026-09-18 06:26:39.709382861 +0000 UTC m=+261.448799032 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b5029a64-fc57-44f7-b2e8-79c9a82c572d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6nf5m" (UID: "b5029a64-fc57-44f7-b2e8-79c9a82c572d") : secret "insights-runtime-extractor-tls" not found Sep 18 06:26:35.922269 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:35.922240 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xccgg_27cccfd2-ff8e-4f71-bc24-5072f09b7f21/node-ca/0.log" Sep 18 06:26:36.321243 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:36.321214 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qxhxp_3320df23-25b7-4e16-9024-076401f7ca3e/serve-healthcheck-canary/0.log" Sep 18 06:26:37.323098 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:37.323066 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-kzkk8_b856fd0d-7ea9-4899-958a-98efec61fe44/kube-storage-version-migrator-operator/0.log" Sep 18 06:26:39.743627 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:39.743590 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b5029a64-fc57-44f7-b2e8-79c9a82c572d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6nf5m\" (UID: \"b5029a64-fc57-44f7-b2e8-79c9a82c572d\") " pod="openshift-insights/insights-runtime-extractor-6nf5m" Sep 18 06:26:39.744145 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:39.743686 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b5029a64-fc57-44f7-b2e8-79c9a82c572d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6nf5m\" (UID: \"b5029a64-fc57-44f7-b2e8-79c9a82c572d\") " pod="openshift-insights/insights-runtime-extractor-6nf5m" Sep 18 06:26:39.744145 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:39.743736 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 18 06:26:39.744145 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:39.743797 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 18 06:26:39.744145 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:39.743801 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5029a64-fc57-44f7-b2e8-79c9a82c572d-insights-runtime-extractor-tls podName:b5029a64-fc57-44f7-b2e8-79c9a82c572d nodeName:}" failed. No retries permitted until 2026-09-18 06:26:47.743786511 +0000 UTC m=+269.483202667 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b5029a64-fc57-44f7-b2e8-79c9a82c572d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6nf5m" (UID: "b5029a64-fc57-44f7-b2e8-79c9a82c572d") : secret "insights-runtime-extractor-tls" not found Sep 18 06:26:39.744145 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:39.743848 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b5029a64-fc57-44f7-b2e8-79c9a82c572d-kube-rbac-proxy-cm podName:b5029a64-fc57-44f7-b2e8-79c9a82c572d nodeName:}" failed. No retries permitted until 2026-09-18 06:26:47.743836255 +0000 UTC m=+269.483252408 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b5029a64-fc57-44f7-b2e8-79c9a82c572d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6nf5m" (UID: "b5029a64-fc57-44f7-b2e8-79c9a82c572d") : configmap "kube-rbac-proxy" not found Sep 18 06:26:42.769288 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:42.769243 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dcc10422-179e-4253-95a4-b8621790f350-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-qcsfv\" (UID: \"dcc10422-179e-4253-95a4-b8621790f350\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-qcsfv" Sep 18 06:26:42.769288 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:42.769291 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e16d183f-ea45-4a4c-94be-f078321c3c79-metrics-certs\") pod \"router-default-75b6d7b44b-fvj8r\" (UID: \"e16d183f-ea45-4a4c-94be-f078321c3c79\") " pod="openshift-ingress/router-default-75b6d7b44b-fvj8r" Sep 18 06:26:42.769759 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:42.769343 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e16d183f-ea45-4a4c-94be-f078321c3c79-service-ca-bundle\") pod \"router-default-75b6d7b44b-fvj8r\" (UID: \"e16d183f-ea45-4a4c-94be-f078321c3c79\") " pod="openshift-ingress/router-default-75b6d7b44b-fvj8r" Sep 18 06:26:42.769759 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:42.769394 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c26181ee-5e97-4934-a7da-71a24e303856-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-tqkh6\" (UID: \"c26181ee-5e97-4934-a7da-71a24e303856\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-tqkh6" Sep 18 06:26:42.769759 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:42.769484 2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 18 06:26:42.769759 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:42.769518 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e16d183f-ea45-4a4c-94be-f078321c3c79-service-ca-bundle podName:e16d183f-ea45-4a4c-94be-f078321c3c79 nodeName:}" failed. No retries permitted until 2026-09-18 06:26:58.769489783 +0000 UTC m=+280.508905957 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e16d183f-ea45-4a4c-94be-f078321c3c79-service-ca-bundle") pod "router-default-75b6d7b44b-fvj8r" (UID: "e16d183f-ea45-4a4c-94be-f078321c3c79") : configmap references non-existent config key: service-ca.crt Sep 18 06:26:42.769759 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:42.769549 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c26181ee-5e97-4934-a7da-71a24e303856-cluster-monitoring-operator-tls podName:c26181ee-5e97-4934-a7da-71a24e303856 nodeName:}" failed. No retries permitted until 2026-09-18 06:26:58.769533576 +0000 UTC m=+280.508949730 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c26181ee-5e97-4934-a7da-71a24e303856-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-tqkh6" (UID: "c26181ee-5e97-4934-a7da-71a24e303856") : secret "cluster-monitoring-operator-tls" not found Sep 18 06:26:42.771764 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:42.771743 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e16d183f-ea45-4a4c-94be-f078321c3c79-metrics-certs\") pod \"router-default-75b6d7b44b-fvj8r\" (UID: \"e16d183f-ea45-4a4c-94be-f078321c3c79\") " pod="openshift-ingress/router-default-75b6d7b44b-fvj8r" Sep 18 06:26:42.771850 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:42.771833 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dcc10422-179e-4253-95a4-b8621790f350-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-qcsfv\" (UID: \"dcc10422-179e-4253-95a4-b8621790f350\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-qcsfv" Sep 18 06:26:42.823341 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:42.823315 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-qcsfv" Sep 18 06:26:42.870387 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:42.870333 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b-registry-tls\") pod \"image-registry-7cfd7d874-4m95j\" (UID: \"ca057b3a-aa6d-45ff-bad8-6dc74bf2009b\") " pod="openshift-image-registry/image-registry-7cfd7d874-4m95j" Sep 18 06:26:42.873832 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:42.873801 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b-registry-tls\") pod \"image-registry-7cfd7d874-4m95j\" (UID: \"ca057b3a-aa6d-45ff-bad8-6dc74bf2009b\") " pod="openshift-image-registry/image-registry-7cfd7d874-4m95j" Sep 18 06:26:42.926147 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:42.926112 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cfd7d874-4m95j" Sep 18 06:26:43.091289 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:43.091261 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-qcsfv"] Sep 18 06:26:43.193907 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:43.193879 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7cfd7d874-4m95j"] Sep 18 06:26:43.196512 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:26:43.196486 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca057b3a_aa6d_45ff_bad8_6dc74bf2009b.slice/crio-eb64e68340cdb2baebfdab222ec8465663b3e81f30f563ea4e7cc24e5aa6f500 WatchSource:0}: Error finding container eb64e68340cdb2baebfdab222ec8465663b3e81f30f563ea4e7cc24e5aa6f500: Status 404 returned error can't find the container with id eb64e68340cdb2baebfdab222ec8465663b3e81f30f563ea4e7cc24e5aa6f500 Sep 18 06:26:43.208953 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:43.208925 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-qcsfv" event={"ID":"dcc10422-179e-4253-95a4-b8621790f350","Type":"ContainerStarted","Data":"4e8d6d29acb6baf9bcfedbde6ae920e23e47c12657c1b84ad581c8a9833e1a37"} Sep 18 06:26:43.210054 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:43.210031 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cfd7d874-4m95j" event={"ID":"ca057b3a-aa6d-45ff-bad8-6dc74bf2009b","Type":"ContainerStarted","Data":"eb64e68340cdb2baebfdab222ec8465663b3e81f30f563ea4e7cc24e5aa6f500"} Sep 18 06:26:44.215608 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:44.215562 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cfd7d874-4m95j" event={"ID":"ca057b3a-aa6d-45ff-bad8-6dc74bf2009b","Type":"ContainerStarted","Data":"cc55630ff59a7966111397073c821db036c6946e759d304f6b222e132cea926f"} Sep 18 06:26:44.216002 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:44.215668 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7cfd7d874-4m95j" Sep 18 06:26:44.233957 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:44.233907 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7cfd7d874-4m95j" podStartSLOduration=18.233892213 podStartE2EDuration="18.233892213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:26:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:26:44.233412435 +0000 UTC m=+265.972828633" watchObservedRunningTime="2026-09-18 06:26:44.233892213 +0000 UTC m=+265.973308384" Sep 18 06:26:45.219962 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:45.219924 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-qcsfv" event={"ID":"dcc10422-179e-4253-95a4-b8621790f350","Type":"ContainerStarted","Data":"d54dc2b9c0f165df7cef21563c05a3e9109dea1f26740d428d662e0997260a3e"} Sep 18 06:26:45.220332 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:45.219978 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-qcsfv" event={"ID":"dcc10422-179e-4253-95a4-b8621790f350","Type":"ContainerStarted","Data":"59c8cdcb80913f1f15e680912fc29b72c7a90c4b716f89f3322d1a48cd85e776"} Sep 18 06:26:45.235860 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:45.235811 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-qcsfv" podStartSLOduration=17.707511796 podStartE2EDuration="19.235796135s" totalImagesPullingTime="1.528284339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:26:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:26:45.233974384 +0000 UTC m=+266.973390587" watchObservedRunningTime="2026-09-18 06:26:45.235796135 +0000 UTC m=+266.975212309" Sep 18 06:26:47.807667 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:47.807633 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b5029a64-fc57-44f7-b2e8-79c9a82c572d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6nf5m\" (UID: \"b5029a64-fc57-44f7-b2e8-79c9a82c572d\") " pod="openshift-insights/insights-runtime-extractor-6nf5m" Sep 18 06:26:47.808037 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:47.807700 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b5029a64-fc57-44f7-b2e8-79c9a82c572d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6nf5m\" (UID: \"b5029a64-fc57-44f7-b2e8-79c9a82c572d\") " pod="openshift-insights/insights-runtime-extractor-6nf5m" Sep 18 06:26:47.808037 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:47.807780 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 18 06:26:47.808037 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:26:47.807848 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5029a64-fc57-44f7-b2e8-79c9a82c572d-insights-runtime-extractor-tls podName:b5029a64-fc57-44f7-b2e8-79c9a82c572d nodeName:}" failed. No retries permitted until 2026-09-18 06:27:03.807833324 +0000 UTC m=+285.547249495 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b5029a64-fc57-44f7-b2e8-79c9a82c572d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6nf5m" (UID: "b5029a64-fc57-44f7-b2e8-79c9a82c572d") : secret "insights-runtime-extractor-tls" not found Sep 18 06:26:47.808182 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:47.808166 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b5029a64-fc57-44f7-b2e8-79c9a82c572d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6nf5m\" (UID: \"b5029a64-fc57-44f7-b2e8-79c9a82c572d\") " pod="openshift-insights/insights-runtime-extractor-6nf5m" Sep 18 06:26:56.413309 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:56.413274 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7cfd7d874-4m95j"] Sep 18 06:26:58.794340 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:58.794005 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e16d183f-ea45-4a4c-94be-f078321c3c79-service-ca-bundle\") pod \"router-default-75b6d7b44b-fvj8r\" (UID: \"e16d183f-ea45-4a4c-94be-f078321c3c79\") " pod="openshift-ingress/router-default-75b6d7b44b-fvj8r" Sep 18 06:26:58.794941 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:58.794911 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c26181ee-5e97-4934-a7da-71a24e303856-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-tqkh6\" (UID: \"c26181ee-5e97-4934-a7da-71a24e303856\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-tqkh6" Sep 18 06:26:58.796399 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:58.795267 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e16d183f-ea45-4a4c-94be-f078321c3c79-service-ca-bundle\") pod \"router-default-75b6d7b44b-fvj8r\" (UID: \"e16d183f-ea45-4a4c-94be-f078321c3c79\") " pod="openshift-ingress/router-default-75b6d7b44b-fvj8r" Sep 18 06:26:58.798602 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:58.798565 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c26181ee-5e97-4934-a7da-71a24e303856-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-tqkh6\" (UID: \"c26181ee-5e97-4934-a7da-71a24e303856\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-tqkh6" Sep 18 06:26:59.031458 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:59.031416 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-tqkh6" Sep 18 06:26:59.043540 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:59.043514 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-75b6d7b44b-fvj8r" Sep 18 06:26:59.302604 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:59.302578 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-tqkh6"] Sep 18 06:26:59.305031 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:26:59.305004 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc26181ee_5e97_4934_a7da_71a24e303856.slice/crio-3abc2d962be085a570211315228c054ba04f923e5b347fa10c3f659d2854d12b WatchSource:0}: Error finding container 3abc2d962be085a570211315228c054ba04f923e5b347fa10c3f659d2854d12b: Status 404 returned error can't find the container with id 3abc2d962be085a570211315228c054ba04f923e5b347fa10c3f659d2854d12b Sep 18 06:26:59.323097 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:26:59.323076 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-75b6d7b44b-fvj8r"] Sep 18 06:26:59.325550 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:26:59.325529 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode16d183f_ea45_4a4c_94be_f078321c3c79.slice/crio-3278faba67fba5cec966af25687e28c411c0bdc86024cd1d7bde05ac460fcfc9 WatchSource:0}: Error finding container 3278faba67fba5cec966af25687e28c411c0bdc86024cd1d7bde05ac460fcfc9: Status 404 returned error can't find the container with id 3278faba67fba5cec966af25687e28c411c0bdc86024cd1d7bde05ac460fcfc9 Sep 18 06:27:00.267776 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:00.267738 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-tqkh6" event={"ID":"c26181ee-5e97-4934-a7da-71a24e303856","Type":"ContainerStarted","Data":"3abc2d962be085a570211315228c054ba04f923e5b347fa10c3f659d2854d12b"} Sep 18 06:27:00.269379 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:00.269336 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-75b6d7b44b-fvj8r" event={"ID":"e16d183f-ea45-4a4c-94be-f078321c3c79","Type":"ContainerStarted","Data":"52b8dea88558b341c250f3dfe909b665ef1091a3c4e36e156440917b8db7b542"} Sep 18 06:27:00.269500 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:00.269406 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-75b6d7b44b-fvj8r" event={"ID":"e16d183f-ea45-4a4c-94be-f078321c3c79","Type":"ContainerStarted","Data":"3278faba67fba5cec966af25687e28c411c0bdc86024cd1d7bde05ac460fcfc9"} Sep 18 06:27:00.288857 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:00.288808 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-75b6d7b44b-fvj8r" podStartSLOduration=34.288791191 podStartE2EDuration="34.288791191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:26:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:27:00.286919967 +0000 UTC m=+282.026336162" watchObservedRunningTime="2026-09-18 06:27:00.288791191 +0000 UTC m=+282.028207367" Sep 18 06:27:00.916745 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:27:00.916705 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-ph66s" podUID="8c9f31bd-99bd-4a50-8d28-72dfc56290ae" Sep 18 06:27:01.044329 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:01.044307 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-75b6d7b44b-fvj8r" Sep 18 06:27:01.047024 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:01.046999 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-75b6d7b44b-fvj8r" Sep 18 06:27:01.274649 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:01.274559 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-tqkh6" event={"ID":"c26181ee-5e97-4934-a7da-71a24e303856","Type":"ContainerStarted","Data":"577c27004760aae7e82bd0e7f2d2654db34be9943e83ddc20ba3ff4028ea1a40"} Sep 18 06:27:01.275075 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:01.274725 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ph66s" Sep 18 06:27:01.275075 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:01.274919 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-75b6d7b44b-fvj8r" Sep 18 06:27:01.276026 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:01.276008 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-75b6d7b44b-fvj8r" Sep 18 06:27:01.289525 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:01.289474 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-tqkh6" podStartSLOduration=33.556326547 podStartE2EDuration="35.28945852s" totalImagesPullingTime="1.733131973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:26:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:27:01.288940354 +0000 UTC m=+283.028356552" watchObservedRunningTime="2026-09-18 06:27:01.28945852 +0000 UTC m=+283.028874695" Sep 18 06:27:03.834883 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:03.834852 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b5029a64-fc57-44f7-b2e8-79c9a82c572d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6nf5m\" (UID: \"b5029a64-fc57-44f7-b2e8-79c9a82c572d\") " pod="openshift-insights/insights-runtime-extractor-6nf5m" Sep 18 06:27:03.837139 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:03.837118 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b5029a64-fc57-44f7-b2e8-79c9a82c572d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6nf5m\" (UID: \"b5029a64-fc57-44f7-b2e8-79c9a82c572d\") " pod="openshift-insights/insights-runtime-extractor-6nf5m" Sep 18 06:27:04.065942 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:04.065915 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6nf5m" Sep 18 06:27:04.362645 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:04.362576 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6nf5m"] Sep 18 06:27:04.365384 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:27:04.365345 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5029a64_fc57_44f7_b2e8_79c9a82c572d.slice/crio-a35e885ebef4560ea0ebc04257ac549e30ab5028876674d154fbcfebb174b8a8 WatchSource:0}: Error finding container a35e885ebef4560ea0ebc04257ac549e30ab5028876674d154fbcfebb174b8a8: Status 404 returned error can't find the container with id a35e885ebef4560ea0ebc04257ac549e30ab5028876674d154fbcfebb174b8a8 Sep 18 06:27:04.640320 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:04.640247 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c9f31bd-99bd-4a50-8d28-72dfc56290ae-metrics-tls\") pod \"dns-default-ph66s\" (UID: \"8c9f31bd-99bd-4a50-8d28-72dfc56290ae\") " pod="openshift-dns/dns-default-ph66s" Sep 18 06:27:04.642460 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:04.642441 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c9f31bd-99bd-4a50-8d28-72dfc56290ae-metrics-tls\") pod \"dns-default-ph66s\" (UID: \"8c9f31bd-99bd-4a50-8d28-72dfc56290ae\") " pod="openshift-dns/dns-default-ph66s" Sep 18 06:27:04.877828 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:04.877794 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-sn4gt\"" Sep 18 06:27:04.885935 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:04.885912 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ph66s" Sep 18 06:27:05.168022 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:05.167953 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ph66s"] Sep 18 06:27:05.170700 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:27:05.170675 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c9f31bd_99bd_4a50_8d28_72dfc56290ae.slice/crio-b1b301160e76bdf8f53297e401dc173a95c0e0c23309d56134e1969076aff6ce WatchSource:0}: Error finding container b1b301160e76bdf8f53297e401dc173a95c0e0c23309d56134e1969076aff6ce: Status 404 returned error can't find the container with id b1b301160e76bdf8f53297e401dc173a95c0e0c23309d56134e1969076aff6ce Sep 18 06:27:05.289089 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:05.289049 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ph66s" event={"ID":"8c9f31bd-99bd-4a50-8d28-72dfc56290ae","Type":"ContainerStarted","Data":"b1b301160e76bdf8f53297e401dc173a95c0e0c23309d56134e1969076aff6ce"} Sep 18 06:27:05.290648 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:05.290627 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6nf5m" event={"ID":"b5029a64-fc57-44f7-b2e8-79c9a82c572d","Type":"ContainerStarted","Data":"ecfccfc51d9c648d798f29b9b432f6287bdddcfc2d9483771c32c07d7d792fb3"} Sep 18 06:27:05.290766 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:05.290667 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6nf5m" event={"ID":"b5029a64-fc57-44f7-b2e8-79c9a82c572d","Type":"ContainerStarted","Data":"b3a8022d24bafd588d7efe8cd6ba29e9e8156c3ddbc138d95689325214055475"} Sep 18 06:27:05.290766 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:05.290685 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6nf5m" event={"ID":"b5029a64-fc57-44f7-b2e8-79c9a82c572d","Type":"ContainerStarted","Data":"a35e885ebef4560ea0ebc04257ac549e30ab5028876674d154fbcfebb174b8a8"} Sep 18 06:27:06.420151 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:06.420122 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7cfd7d874-4m95j" Sep 18 06:27:07.299529 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:07.299494 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ph66s" event={"ID":"8c9f31bd-99bd-4a50-8d28-72dfc56290ae","Type":"ContainerStarted","Data":"cabd7865c4705b4eaaf00157c6c80def8f438f98966655c6a99723a1a8a39836"} Sep 18 06:27:07.299780 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:07.299548 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ph66s" event={"ID":"8c9f31bd-99bd-4a50-8d28-72dfc56290ae","Type":"ContainerStarted","Data":"637526f8806cde0cd57f53ea7b32f8fedf5711bc396999d994756c2cfda4a363"} Sep 18 06:27:07.299780 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:07.299575 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-ph66s" Sep 18 06:27:07.301766 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:07.301745 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6nf5m" event={"ID":"b5029a64-fc57-44f7-b2e8-79c9a82c572d","Type":"ContainerStarted","Data":"4a547af7fa7ac0e7cf0551242cc0b08de90dfe9eb88f69097073cee74cde8238"} Sep 18 06:27:07.316353 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:07.316310 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-ph66s" podStartSLOduration=251.760832083 podStartE2EDuration="4m13.316298027s" totalImagesPullingTime="1.555465944s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:22:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:27:07.315131188 +0000 UTC m=+289.054547388" watchObservedRunningTime="2026-09-18 06:27:07.316298027 +0000 UTC m=+289.055714201" Sep 18 06:27:07.332333 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:07.332292 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-6nf5m" podStartSLOduration=34.137329549 podStartE2EDuration="36.332279551s" totalImagesPullingTime="2.194950002s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 06:26:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:27:07.331859158 +0000 UTC m=+289.071275332" watchObservedRunningTime="2026-09-18 06:27:07.332279551 +0000 UTC m=+289.071695725" Sep 18 06:27:09.923818 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:09.922203 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-24gd5"] Sep 18 06:27:09.926198 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:09.926172 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-24gd5" Sep 18 06:27:09.926534 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:09.926503 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-4rqjv"] Sep 18 06:27:09.928526 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:09.928503 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Sep 18 06:27:09.929344 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:09.929326 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-4rqjv" Sep 18 06:27:09.929592 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:09.929568 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 18 06:27:09.929655 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:09.929568 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-dxq8q\"" Sep 18 06:27:09.929825 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:09.929810 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Sep 18 06:27:09.931298 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:09.931282 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 18 06:27:09.931846 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:09.931830 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 18 06:27:09.932007 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:09.931986 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-5hh7z\"" Sep 18 06:27:09.932065 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:09.932005 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 18 06:27:09.937247 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:09.937219 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-24gd5"] Sep 18 06:27:09.943829 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:09.943808 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-4rqjv"] Sep 18 06:27:09.944660 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:09.944640 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-nzscf"] Sep 18 06:27:09.947313 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:09.947294 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nzscf" Sep 18 06:27:09.949508 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:09.949488 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-cmnll\"" Sep 18 06:27:09.949622 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:09.949607 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 18 06:27:09.949704 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:09.949689 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 18 06:27:09.950059 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:09.950042 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 18 06:27:09.979330 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:09.979306 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3fb98b73-fc76-4fab-928c-8c60bb29354c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-24gd5\" (UID: \"3fb98b73-fc76-4fab-928c-8c60bb29354c\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-24gd5" Sep 18 06:27:09.979330 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:09.979331 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b642bccf-1547-424f-95ea-6de9749953a1-sys\") pod \"node-exporter-nzscf\" (UID: \"b642bccf-1547-424f-95ea-6de9749953a1\") " pod="openshift-monitoring/node-exporter-nzscf" Sep 18 06:27:09.979489 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:09.979347 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b642bccf-1547-424f-95ea-6de9749953a1-root\") pod \"node-exporter-nzscf\" (UID: \"b642bccf-1547-424f-95ea-6de9749953a1\") " pod="openshift-monitoring/node-exporter-nzscf" Sep 18 06:27:09.979489 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:09.979379 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b642bccf-1547-424f-95ea-6de9749953a1-node-exporter-textfile\") pod \"node-exporter-nzscf\" (UID: \"b642bccf-1547-424f-95ea-6de9749953a1\") " pod="openshift-monitoring/node-exporter-nzscf" Sep 18 06:27:09.979489 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:09.979409 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3b9180cd-aece-4e1a-bd4f-781d7dd5ebf9-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-4rqjv\" (UID: \"3b9180cd-aece-4e1a-bd4f-781d7dd5ebf9\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-4rqjv" Sep 18 06:27:09.979489 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:09.979451 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3fb98b73-fc76-4fab-928c-8c60bb29354c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-24gd5\" (UID: \"3fb98b73-fc76-4fab-928c-8c60bb29354c\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-24gd5" Sep 18 06:27:09.979489 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:09.979481 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3b9180cd-aece-4e1a-bd4f-781d7dd5ebf9-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-4rqjv\" (UID: \"3b9180cd-aece-4e1a-bd4f-781d7dd5ebf9\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-4rqjv" Sep 18 06:27:09.979652 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:09.979512 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ch5g\" (UniqueName: \"kubernetes.io/projected/b642bccf-1547-424f-95ea-6de9749953a1-kube-api-access-9ch5g\") pod \"node-exporter-nzscf\" (UID: \"b642bccf-1547-424f-95ea-6de9749953a1\") " pod="openshift-monitoring/node-exporter-nzscf" Sep 18 06:27:09.979652 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:09.979574 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b642bccf-1547-424f-95ea-6de9749953a1-node-exporter-tls\") pod \"node-exporter-nzscf\" (UID: \"b642bccf-1547-424f-95ea-6de9749953a1\") " pod="openshift-monitoring/node-exporter-nzscf" Sep 18 06:27:09.979652 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:09.979620 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b642bccf-1547-424f-95ea-6de9749953a1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nzscf\" (UID: \"b642bccf-1547-424f-95ea-6de9749953a1\") " pod="openshift-monitoring/node-exporter-nzscf" Sep 18 06:27:09.979652 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:09.979644 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b642bccf-1547-424f-95ea-6de9749953a1-node-exporter-wtmp\") pod \"node-exporter-nzscf\" (UID: \"b642bccf-1547-424f-95ea-6de9749953a1\") " pod="openshift-monitoring/node-exporter-nzscf" Sep 18 06:27:09.979786 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:09.979670 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b642bccf-1547-424f-95ea-6de9749953a1-metrics-client-ca\") pod \"node-exporter-nzscf\" (UID: \"b642bccf-1547-424f-95ea-6de9749953a1\") " pod="openshift-monitoring/node-exporter-nzscf" Sep 18 06:27:09.979786 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:09.979695 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vppsx\" (UniqueName: \"kubernetes.io/projected/3fb98b73-fc76-4fab-928c-8c60bb29354c-kube-api-access-vppsx\") pod \"openshift-state-metrics-86c5bc79d6-24gd5\" (UID: \"3fb98b73-fc76-4fab-928c-8c60bb29354c\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-24gd5" Sep 18 06:27:09.979786 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:09.979710 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt5kf\" (UniqueName: \"kubernetes.io/projected/3b9180cd-aece-4e1a-bd4f-781d7dd5ebf9-kube-api-access-xt5kf\") pod \"kube-state-metrics-84c885658c-4rqjv\" (UID: \"3b9180cd-aece-4e1a-bd4f-781d7dd5ebf9\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-4rqjv" Sep 18 06:27:09.979786 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:09.979726 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b642bccf-1547-424f-95ea-6de9749953a1-node-exporter-accelerators-collector-config\") pod \"node-exporter-nzscf\" (UID: \"b642bccf-1547-424f-95ea-6de9749953a1\") " pod="openshift-monitoring/node-exporter-nzscf" Sep 18 06:27:09.979786 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:09.979746 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3b9180cd-aece-4e1a-bd4f-781d7dd5ebf9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-4rqjv\" (UID: \"3b9180cd-aece-4e1a-bd4f-781d7dd5ebf9\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-4rqjv" Sep 18 06:27:09.979944 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:09.979800 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3fb98b73-fc76-4fab-928c-8c60bb29354c-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-24gd5\" (UID: \"3fb98b73-fc76-4fab-928c-8c60bb29354c\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-24gd5" Sep 18 06:27:09.979944 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:09.979824 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b9180cd-aece-4e1a-bd4f-781d7dd5ebf9-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-4rqjv\" (UID: \"3b9180cd-aece-4e1a-bd4f-781d7dd5ebf9\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-4rqjv" Sep 18 06:27:09.979944 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:09.979855 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3b9180cd-aece-4e1a-bd4f-781d7dd5ebf9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-4rqjv\" (UID: \"3b9180cd-aece-4e1a-bd4f-781d7dd5ebf9\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-4rqjv" Sep 18 06:27:10.080810 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:10.080773 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b9180cd-aece-4e1a-bd4f-781d7dd5ebf9-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-4rqjv\" (UID: \"3b9180cd-aece-4e1a-bd4f-781d7dd5ebf9\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-4rqjv" Sep 18 06:27:10.080810 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:10.080810 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3b9180cd-aece-4e1a-bd4f-781d7dd5ebf9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-4rqjv\" (UID: \"3b9180cd-aece-4e1a-bd4f-781d7dd5ebf9\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-4rqjv" Sep 18 06:27:10.081020 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:10.080827 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3fb98b73-fc76-4fab-928c-8c60bb29354c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-24gd5\" (UID: \"3fb98b73-fc76-4fab-928c-8c60bb29354c\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-24gd5" Sep 18 06:27:10.081020 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:10.080842 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b642bccf-1547-424f-95ea-6de9749953a1-sys\") pod \"node-exporter-nzscf\" (UID: \"b642bccf-1547-424f-95ea-6de9749953a1\") " pod="openshift-monitoring/node-exporter-nzscf" Sep 18 06:27:10.081020 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:10.080864 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b642bccf-1547-424f-95ea-6de9749953a1-root\") pod \"node-exporter-nzscf\" (UID: \"b642bccf-1547-424f-95ea-6de9749953a1\") " pod="openshift-monitoring/node-exporter-nzscf" Sep 18 06:27:10.081020 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:10.080886 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b642bccf-1547-424f-95ea-6de9749953a1-node-exporter-textfile\") pod \"node-exporter-nzscf\" (UID: \"b642bccf-1547-424f-95ea-6de9749953a1\") " pod="openshift-monitoring/node-exporter-nzscf" Sep 18 06:27:10.081020 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:10.080907 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3b9180cd-aece-4e1a-bd4f-781d7dd5ebf9-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-4rqjv\" (UID: \"3b9180cd-aece-4e1a-bd4f-781d7dd5ebf9\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-4rqjv" Sep 18 06:27:10.081020 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:10.080911 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b642bccf-1547-424f-95ea-6de9749953a1-sys\") pod \"node-exporter-nzscf\" (UID: \"b642bccf-1547-424f-95ea-6de9749953a1\") " pod="openshift-monitoring/node-exporter-nzscf" Sep 18 06:27:10.081020 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:10.080926 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3fb98b73-fc76-4fab-928c-8c60bb29354c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-24gd5\" (UID: \"3fb98b73-fc76-4fab-928c-8c60bb29354c\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-24gd5" Sep 18 06:27:10.081020 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:10.080971 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3b9180cd-aece-4e1a-bd4f-781d7dd5ebf9-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-4rqjv\" (UID: \"3b9180cd-aece-4e1a-bd4f-781d7dd5ebf9\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-4rqjv" Sep 18 06:27:10.081020 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:10.081023 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ch5g\" (UniqueName: \"kubernetes.io/projected/b642bccf-1547-424f-95ea-6de9749953a1-kube-api-access-9ch5g\") pod \"node-exporter-nzscf\" (UID: \"b642bccf-1547-424f-95ea-6de9749953a1\") " pod="openshift-monitoring/node-exporter-nzscf" Sep 18 06:27:10.081460 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:27:10.081026 2567 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Sep 18 06:27:10.081460 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:10.081066 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b642bccf-1547-424f-95ea-6de9749953a1-node-exporter-tls\") pod \"node-exporter-nzscf\" (UID: \"b642bccf-1547-424f-95ea-6de9749953a1\") " pod="openshift-monitoring/node-exporter-nzscf" Sep 18 06:27:10.081460 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:27:10.081085 2567 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Sep 18 06:27:10.081460 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:10.081095 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b642bccf-1547-424f-95ea-6de9749953a1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nzscf\" (UID: \"b642bccf-1547-424f-95ea-6de9749953a1\") " pod="openshift-monitoring/node-exporter-nzscf" Sep 18 06:27:10.081460 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:27:10.081177 2567 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 18 06:27:10.081460 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:10.081268 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b642bccf-1547-424f-95ea-6de9749953a1-node-exporter-textfile\") pod \"node-exporter-nzscf\" (UID: \"b642bccf-1547-424f-95ea-6de9749953a1\") " pod="openshift-monitoring/node-exporter-nzscf" Sep 18 06:27:10.081460 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:10.081345 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b642bccf-1547-424f-95ea-6de9749953a1-root\") pod \"node-exporter-nzscf\" (UID: \"b642bccf-1547-424f-95ea-6de9749953a1\") " pod="openshift-monitoring/node-exporter-nzscf" Sep 18 06:27:10.081460 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:10.081412 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3b9180cd-aece-4e1a-bd4f-781d7dd5ebf9-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-4rqjv\" (UID: \"3b9180cd-aece-4e1a-bd4f-781d7dd5ebf9\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-4rqjv" Sep 18 06:27:10.081768 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:27:10.081544 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fb98b73-fc76-4fab-928c-8c60bb29354c-openshift-state-metrics-tls podName:3fb98b73-fc76-4fab-928c-8c60bb29354c nodeName:}" failed. No retries permitted until 2026-09-18 06:27:10.581112124 +0000 UTC m=+292.320528295 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/3fb98b73-fc76-4fab-928c-8c60bb29354c-openshift-state-metrics-tls") pod "openshift-state-metrics-86c5bc79d6-24gd5" (UID: "3fb98b73-fc76-4fab-928c-8c60bb29354c") : secret "openshift-state-metrics-tls" not found Sep 18 06:27:10.081768 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:10.081590 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b642bccf-1547-424f-95ea-6de9749953a1-node-exporter-wtmp\") pod \"node-exporter-nzscf\" (UID: \"b642bccf-1547-424f-95ea-6de9749953a1\") " pod="openshift-monitoring/node-exporter-nzscf" Sep 18 06:27:10.081768 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:10.081637 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b642bccf-1547-424f-95ea-6de9749953a1-metrics-client-ca\") pod \"node-exporter-nzscf\" (UID: \"b642bccf-1547-424f-95ea-6de9749953a1\") " pod="openshift-monitoring/node-exporter-nzscf" Sep 18 06:27:10.081768 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:10.081664 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vppsx\" (UniqueName: \"kubernetes.io/projected/3fb98b73-fc76-4fab-928c-8c60bb29354c-kube-api-access-vppsx\") pod \"openshift-state-metrics-86c5bc79d6-24gd5\" (UID: \"3fb98b73-fc76-4fab-928c-8c60bb29354c\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-24gd5" Sep 18 06:27:10.081768 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:10.081695 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b642bccf-1547-424f-95ea-6de9749953a1-node-exporter-wtmp\") pod \"node-exporter-nzscf\" (UID: \"b642bccf-1547-424f-95ea-6de9749953a1\") " pod="openshift-monitoring/node-exporter-nzscf" Sep 18 06:27:10.081768 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:10.081751 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b9180cd-aece-4e1a-bd4f-781d7dd5ebf9-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-4rqjv\" (UID: \"3b9180cd-aece-4e1a-bd4f-781d7dd5ebf9\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-4rqjv" Sep 18 06:27:10.082081 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:10.081794 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt5kf\" (UniqueName: \"kubernetes.io/projected/3b9180cd-aece-4e1a-bd4f-781d7dd5ebf9-kube-api-access-xt5kf\") pod \"kube-state-metrics-84c885658c-4rqjv\" (UID: \"3b9180cd-aece-4e1a-bd4f-781d7dd5ebf9\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-4rqjv" Sep 18 06:27:10.082081 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:10.081832 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b642bccf-1547-424f-95ea-6de9749953a1-node-exporter-accelerators-collector-config\") pod \"node-exporter-nzscf\" (UID: \"b642bccf-1547-424f-95ea-6de9749953a1\") " pod="openshift-monitoring/node-exporter-nzscf" Sep 18 06:27:10.082081 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:10.081867 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3b9180cd-aece-4e1a-bd4f-781d7dd5ebf9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-4rqjv\" (UID: \"3b9180cd-aece-4e1a-bd4f-781d7dd5ebf9\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-4rqjv" Sep 18 06:27:10.082081 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:10.081927 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3fb98b73-fc76-4fab-928c-8c60bb29354c-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-24gd5\" (UID: \"3fb98b73-fc76-4fab-928c-8c60bb29354c\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-24gd5" Sep 18 06:27:10.082277 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:10.082151 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b642bccf-1547-424f-95ea-6de9749953a1-metrics-client-ca\") pod \"node-exporter-nzscf\" (UID: \"b642bccf-1547-424f-95ea-6de9749953a1\") " pod="openshift-monitoring/node-exporter-nzscf" Sep 18 06:27:10.082277 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:27:10.082212 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b9180cd-aece-4e1a-bd4f-781d7dd5ebf9-kube-state-metrics-tls podName:3b9180cd-aece-4e1a-bd4f-781d7dd5ebf9 nodeName:}" failed. No retries permitted until 2026-09-18 06:27:10.582195025 +0000 UTC m=+292.321611177 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/3b9180cd-aece-4e1a-bd4f-781d7dd5ebf9-kube-state-metrics-tls") pod "kube-state-metrics-84c885658c-4rqjv" (UID: "3b9180cd-aece-4e1a-bd4f-781d7dd5ebf9") : secret "kube-state-metrics-tls" not found Sep 18 06:27:10.082277 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:27:10.082249 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b642bccf-1547-424f-95ea-6de9749953a1-node-exporter-tls podName:b642bccf-1547-424f-95ea-6de9749953a1 nodeName:}" failed. No retries permitted until 2026-09-18 06:27:10.582236966 +0000 UTC m=+292.321653118 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/b642bccf-1547-424f-95ea-6de9749953a1-node-exporter-tls") pod "node-exporter-nzscf" (UID: "b642bccf-1547-424f-95ea-6de9749953a1") : secret "node-exporter-tls" not found Sep 18 06:27:10.082455 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:10.082395 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b642bccf-1547-424f-95ea-6de9749953a1-node-exporter-accelerators-collector-config\") pod \"node-exporter-nzscf\" (UID: \"b642bccf-1547-424f-95ea-6de9749953a1\") " pod="openshift-monitoring/node-exporter-nzscf" Sep 18 06:27:10.082602 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:10.082584 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3fb98b73-fc76-4fab-928c-8c60bb29354c-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-24gd5\" (UID: \"3fb98b73-fc76-4fab-928c-8c60bb29354c\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-24gd5" Sep 18 06:27:10.082857 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:10.082832 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3b9180cd-aece-4e1a-bd4f-781d7dd5ebf9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-4rqjv\" (UID: \"3b9180cd-aece-4e1a-bd4f-781d7dd5ebf9\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-4rqjv" Sep 18 06:27:10.083745 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:10.083719 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3b9180cd-aece-4e1a-bd4f-781d7dd5ebf9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-4rqjv\" (UID: \"3b9180cd-aece-4e1a-bd4f-781d7dd5ebf9\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-4rqjv" Sep 18 06:27:10.084080 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:10.084059 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b642bccf-1547-424f-95ea-6de9749953a1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nzscf\" (UID: \"b642bccf-1547-424f-95ea-6de9749953a1\") " pod="openshift-monitoring/node-exporter-nzscf" Sep 18 06:27:10.084150 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:10.084096 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3fb98b73-fc76-4fab-928c-8c60bb29354c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-24gd5\" (UID: \"3fb98b73-fc76-4fab-928c-8c60bb29354c\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-24gd5" Sep 18 06:27:10.088577 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:10.088561 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ch5g\" (UniqueName: \"kubernetes.io/projected/b642bccf-1547-424f-95ea-6de9749953a1-kube-api-access-9ch5g\") pod \"node-exporter-nzscf\" (UID: \"b642bccf-1547-424f-95ea-6de9749953a1\") " pod="openshift-monitoring/node-exporter-nzscf" Sep 18 06:27:10.092054 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:10.092032 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt5kf\" (UniqueName: \"kubernetes.io/projected/3b9180cd-aece-4e1a-bd4f-781d7dd5ebf9-kube-api-access-xt5kf\") pod \"kube-state-metrics-84c885658c-4rqjv\" (UID: \"3b9180cd-aece-4e1a-bd4f-781d7dd5ebf9\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-4rqjv" Sep 18 06:27:10.094764 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:10.094742 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vppsx\" (UniqueName: \"kubernetes.io/projected/3fb98b73-fc76-4fab-928c-8c60bb29354c-kube-api-access-vppsx\") pod \"openshift-state-metrics-86c5bc79d6-24gd5\" (UID: \"3fb98b73-fc76-4fab-928c-8c60bb29354c\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-24gd5" Sep 18 06:27:10.586271 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:10.586220 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3fb98b73-fc76-4fab-928c-8c60bb29354c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-24gd5\" (UID: \"3fb98b73-fc76-4fab-928c-8c60bb29354c\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-24gd5" Sep 18 06:27:10.586271 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:10.586279 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3b9180cd-aece-4e1a-bd4f-781d7dd5ebf9-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-4rqjv\" (UID: \"3b9180cd-aece-4e1a-bd4f-781d7dd5ebf9\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-4rqjv" Sep 18 06:27:10.586677 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:10.586324 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b642bccf-1547-424f-95ea-6de9749953a1-node-exporter-tls\") pod \"node-exporter-nzscf\" (UID: \"b642bccf-1547-424f-95ea-6de9749953a1\") " pod="openshift-monitoring/node-exporter-nzscf" Sep 18 06:27:10.586677 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:27:10.586409 2567 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Sep 18 06:27:10.586677 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:27:10.586498 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fb98b73-fc76-4fab-928c-8c60bb29354c-openshift-state-metrics-tls podName:3fb98b73-fc76-4fab-928c-8c60bb29354c nodeName:}" failed. No retries permitted until 2026-09-18 06:27:11.586478331 +0000 UTC m=+293.325894489 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/3fb98b73-fc76-4fab-928c-8c60bb29354c-openshift-state-metrics-tls") pod "openshift-state-metrics-86c5bc79d6-24gd5" (UID: "3fb98b73-fc76-4fab-928c-8c60bb29354c") : secret "openshift-state-metrics-tls" not found Sep 18 06:27:10.588750 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:10.588724 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b642bccf-1547-424f-95ea-6de9749953a1-node-exporter-tls\") pod \"node-exporter-nzscf\" (UID: \"b642bccf-1547-424f-95ea-6de9749953a1\") " pod="openshift-monitoring/node-exporter-nzscf" Sep 18 06:27:10.588856 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:10.588779 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3b9180cd-aece-4e1a-bd4f-781d7dd5ebf9-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-4rqjv\" (UID: \"3b9180cd-aece-4e1a-bd4f-781d7dd5ebf9\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-4rqjv" Sep 18 06:27:10.841642 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:10.841556 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-4rqjv" Sep 18 06:27:10.857672 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:10.857647 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nzscf" Sep 18 06:27:10.869156 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:27:10.869120 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb642bccf_1547_424f_95ea_6de9749953a1.slice/crio-8060f11a2790659594d3cfe2229ea965564aa2cb3624510d38e43e9a29c0d1d7 WatchSource:0}: Error finding container 8060f11a2790659594d3cfe2229ea965564aa2cb3624510d38e43e9a29c0d1d7: Status 404 returned error can't find the container with id 8060f11a2790659594d3cfe2229ea965564aa2cb3624510d38e43e9a29c0d1d7 Sep 18 06:27:11.118217 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:11.118147 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-4rqjv"] Sep 18 06:27:11.121298 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:27:11.121272 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b9180cd_aece_4e1a_bd4f_781d7dd5ebf9.slice/crio-4b4c58a2a1a010ed573370c4892bc5a1f12e186c5465991b78f0e76478415045 WatchSource:0}: Error finding container 4b4c58a2a1a010ed573370c4892bc5a1f12e186c5465991b78f0e76478415045: Status 404 returned error can't find the container with id 4b4c58a2a1a010ed573370c4892bc5a1f12e186c5465991b78f0e76478415045 Sep 18 06:27:11.316620 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:11.316553 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nzscf" event={"ID":"b642bccf-1547-424f-95ea-6de9749953a1","Type":"ContainerStarted","Data":"8060f11a2790659594d3cfe2229ea965564aa2cb3624510d38e43e9a29c0d1d7"} Sep 18 06:27:11.317899 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:11.317875 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-4rqjv" event={"ID":"3b9180cd-aece-4e1a-bd4f-781d7dd5ebf9","Type":"ContainerStarted","Data":"4b4c58a2a1a010ed573370c4892bc5a1f12e186c5465991b78f0e76478415045"} Sep 18 06:27:11.594641 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:11.594600 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3fb98b73-fc76-4fab-928c-8c60bb29354c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-24gd5\" (UID: \"3fb98b73-fc76-4fab-928c-8c60bb29354c\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-24gd5" Sep 18 06:27:11.597443 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:11.597422 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3fb98b73-fc76-4fab-928c-8c60bb29354c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-24gd5\" (UID: \"3fb98b73-fc76-4fab-928c-8c60bb29354c\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-24gd5" Sep 18 06:27:11.736946 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:11.736891 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-24gd5" Sep 18 06:27:12.184402 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:12.184346 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-24gd5"] Sep 18 06:27:12.323513 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:12.323475 2567 generic.go:356] "Generic (PLEG): container finished" podID="b642bccf-1547-424f-95ea-6de9749953a1" containerID="1305dc647815711044b6215fb2b9febb723daaeb921cefe0cecf511699af777c" exitCode=0 Sep 18 06:27:12.323709 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:12.323537 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nzscf" event={"ID":"b642bccf-1547-424f-95ea-6de9749953a1","Type":"ContainerDied","Data":"1305dc647815711044b6215fb2b9febb723daaeb921cefe0cecf511699af777c"} Sep 18 06:27:12.429733 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:27:12.429704 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fb98b73_fc76_4fab_928c_8c60bb29354c.slice/crio-deccb18b26f9a23bf2a46fce454abec8e795ab798e6d707651eaad0defa34d47 WatchSource:0}: Error finding container deccb18b26f9a23bf2a46fce454abec8e795ab798e6d707651eaad0defa34d47: Status 404 returned error can't find the container with id deccb18b26f9a23bf2a46fce454abec8e795ab798e6d707651eaad0defa34d47 Sep 18 06:27:12.872404 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:12.872355 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-67d649cbb7-jbfsd"] Sep 18 06:27:12.875996 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:12.875976 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-67d649cbb7-jbfsd" Sep 18 06:27:12.878380 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:12.878343 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 18 06:27:12.878637 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:12.878621 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 18 06:27:12.878841 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:12.878819 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-f4dh893m0ntho\"" Sep 18 06:27:12.878841 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:12.878826 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 18 06:27:12.879029 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:12.878864 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 18 06:27:12.879029 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:12.878887 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 18 06:27:12.879029 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:12.878864 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-jmhcl\"" Sep 18 06:27:12.888631 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:12.888610 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-67d649cbb7-jbfsd"] Sep 18 06:27:12.906754 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:12.906720 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5c0016b-5ad1-46f0-9460-ae6207e9d171-metrics-client-ca\") pod \"thanos-querier-67d649cbb7-jbfsd\" (UID: \"d5c0016b-5ad1-46f0-9460-ae6207e9d171\") " pod="openshift-monitoring/thanos-querier-67d649cbb7-jbfsd" Sep 18 06:27:12.906986 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:12.906792 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d5c0016b-5ad1-46f0-9460-ae6207e9d171-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-67d649cbb7-jbfsd\" (UID: \"d5c0016b-5ad1-46f0-9460-ae6207e9d171\") " pod="openshift-monitoring/thanos-querier-67d649cbb7-jbfsd" Sep 18 06:27:12.906986 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:12.906822 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4658\" (UniqueName: \"kubernetes.io/projected/d5c0016b-5ad1-46f0-9460-ae6207e9d171-kube-api-access-w4658\") pod \"thanos-querier-67d649cbb7-jbfsd\" (UID: \"d5c0016b-5ad1-46f0-9460-ae6207e9d171\") " pod="openshift-monitoring/thanos-querier-67d649cbb7-jbfsd" Sep 18 06:27:12.906986 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:12.906869 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d5c0016b-5ad1-46f0-9460-ae6207e9d171-secret-grpc-tls\") pod \"thanos-querier-67d649cbb7-jbfsd\" (UID: \"d5c0016b-5ad1-46f0-9460-ae6207e9d171\") " pod="openshift-monitoring/thanos-querier-67d649cbb7-jbfsd" Sep 18 06:27:12.906986 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:12.906896 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d5c0016b-5ad1-46f0-9460-ae6207e9d171-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-67d649cbb7-jbfsd\" (UID: \"d5c0016b-5ad1-46f0-9460-ae6207e9d171\") " pod="openshift-monitoring/thanos-querier-67d649cbb7-jbfsd" Sep 18 06:27:12.906986 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:12.906926 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d5c0016b-5ad1-46f0-9460-ae6207e9d171-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-67d649cbb7-jbfsd\" (UID: \"d5c0016b-5ad1-46f0-9460-ae6207e9d171\") " pod="openshift-monitoring/thanos-querier-67d649cbb7-jbfsd" Sep 18 06:27:12.906986 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:12.906982 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d5c0016b-5ad1-46f0-9460-ae6207e9d171-secret-thanos-querier-tls\") pod \"thanos-querier-67d649cbb7-jbfsd\" (UID: \"d5c0016b-5ad1-46f0-9460-ae6207e9d171\") " pod="openshift-monitoring/thanos-querier-67d649cbb7-jbfsd" Sep 18 06:27:12.907292 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:12.907013 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d5c0016b-5ad1-46f0-9460-ae6207e9d171-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-67d649cbb7-jbfsd\" (UID: \"d5c0016b-5ad1-46f0-9460-ae6207e9d171\") " pod="openshift-monitoring/thanos-querier-67d649cbb7-jbfsd" Sep 18 06:27:13.008472 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:13.008437 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d5c0016b-5ad1-46f0-9460-ae6207e9d171-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-67d649cbb7-jbfsd\" (UID: \"d5c0016b-5ad1-46f0-9460-ae6207e9d171\") " pod="openshift-monitoring/thanos-querier-67d649cbb7-jbfsd" Sep 18 06:27:13.008671 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:13.008471 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4658\" (UniqueName: \"kubernetes.io/projected/d5c0016b-5ad1-46f0-9460-ae6207e9d171-kube-api-access-w4658\") pod \"thanos-querier-67d649cbb7-jbfsd\" (UID: \"d5c0016b-5ad1-46f0-9460-ae6207e9d171\") " pod="openshift-monitoring/thanos-querier-67d649cbb7-jbfsd" Sep 18 06:27:13.008671 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:13.008508 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d5c0016b-5ad1-46f0-9460-ae6207e9d171-secret-grpc-tls\") pod \"thanos-querier-67d649cbb7-jbfsd\" (UID: \"d5c0016b-5ad1-46f0-9460-ae6207e9d171\") " pod="openshift-monitoring/thanos-querier-67d649cbb7-jbfsd" Sep 18 06:27:13.008671 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:13.008525 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d5c0016b-5ad1-46f0-9460-ae6207e9d171-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-67d649cbb7-jbfsd\" (UID: \"d5c0016b-5ad1-46f0-9460-ae6207e9d171\") " pod="openshift-monitoring/thanos-querier-67d649cbb7-jbfsd" Sep 18 06:27:13.008671 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:13.008550 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d5c0016b-5ad1-46f0-9460-ae6207e9d171-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-67d649cbb7-jbfsd\" (UID: \"d5c0016b-5ad1-46f0-9460-ae6207e9d171\") " pod="openshift-monitoring/thanos-querier-67d649cbb7-jbfsd" Sep 18 06:27:13.008671 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:13.008604 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d5c0016b-5ad1-46f0-9460-ae6207e9d171-secret-thanos-querier-tls\") pod \"thanos-querier-67d649cbb7-jbfsd\" (UID: \"d5c0016b-5ad1-46f0-9460-ae6207e9d171\") " pod="openshift-monitoring/thanos-querier-67d649cbb7-jbfsd" Sep 18 06:27:13.008671 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:13.008667 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d5c0016b-5ad1-46f0-9460-ae6207e9d171-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-67d649cbb7-jbfsd\" (UID: \"d5c0016b-5ad1-46f0-9460-ae6207e9d171\") " pod="openshift-monitoring/thanos-querier-67d649cbb7-jbfsd" Sep 18 06:27:13.008941 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:13.008724 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5c0016b-5ad1-46f0-9460-ae6207e9d171-metrics-client-ca\") pod \"thanos-querier-67d649cbb7-jbfsd\" (UID: \"d5c0016b-5ad1-46f0-9460-ae6207e9d171\") " pod="openshift-monitoring/thanos-querier-67d649cbb7-jbfsd" Sep 18 06:27:13.009569 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:13.009492 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5c0016b-5ad1-46f0-9460-ae6207e9d171-metrics-client-ca\") pod \"thanos-querier-67d649cbb7-jbfsd\" (UID: \"d5c0016b-5ad1-46f0-9460-ae6207e9d171\") " pod="openshift-monitoring/thanos-querier-67d649cbb7-jbfsd" Sep 18 06:27:13.011192 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:13.011170 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d5c0016b-5ad1-46f0-9460-ae6207e9d171-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-67d649cbb7-jbfsd\" (UID: \"d5c0016b-5ad1-46f0-9460-ae6207e9d171\") " pod="openshift-monitoring/thanos-querier-67d649cbb7-jbfsd" Sep 18 06:27:13.011294 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:13.011212 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d5c0016b-5ad1-46f0-9460-ae6207e9d171-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-67d649cbb7-jbfsd\" (UID: \"d5c0016b-5ad1-46f0-9460-ae6207e9d171\") " pod="openshift-monitoring/thanos-querier-67d649cbb7-jbfsd" Sep 18 06:27:13.011501 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:13.011485 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d5c0016b-5ad1-46f0-9460-ae6207e9d171-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-67d649cbb7-jbfsd\" (UID: \"d5c0016b-5ad1-46f0-9460-ae6207e9d171\") " pod="openshift-monitoring/thanos-querier-67d649cbb7-jbfsd" Sep 18 06:27:13.011603 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:13.011584 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d5c0016b-5ad1-46f0-9460-ae6207e9d171-secret-grpc-tls\") pod \"thanos-querier-67d649cbb7-jbfsd\" (UID: \"d5c0016b-5ad1-46f0-9460-ae6207e9d171\") " pod="openshift-monitoring/thanos-querier-67d649cbb7-jbfsd" Sep 18 06:27:13.011603 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:13.011592 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d5c0016b-5ad1-46f0-9460-ae6207e9d171-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-67d649cbb7-jbfsd\" (UID: \"d5c0016b-5ad1-46f0-9460-ae6207e9d171\") " pod="openshift-monitoring/thanos-querier-67d649cbb7-jbfsd" Sep 18 06:27:13.011768 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:13.011754 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d5c0016b-5ad1-46f0-9460-ae6207e9d171-secret-thanos-querier-tls\") pod \"thanos-querier-67d649cbb7-jbfsd\" (UID: \"d5c0016b-5ad1-46f0-9460-ae6207e9d171\") " pod="openshift-monitoring/thanos-querier-67d649cbb7-jbfsd" Sep 18 06:27:13.015665 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:13.015650 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4658\" (UniqueName: \"kubernetes.io/projected/d5c0016b-5ad1-46f0-9460-ae6207e9d171-kube-api-access-w4658\") pod \"thanos-querier-67d649cbb7-jbfsd\" (UID: \"d5c0016b-5ad1-46f0-9460-ae6207e9d171\") " pod="openshift-monitoring/thanos-querier-67d649cbb7-jbfsd" Sep 18 06:27:13.185348 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:13.185254 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-67d649cbb7-jbfsd" Sep 18 06:27:13.331205 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:13.331148 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nzscf" event={"ID":"b642bccf-1547-424f-95ea-6de9749953a1","Type":"ContainerStarted","Data":"aacfecca9f5a9c06dbb3efa3346814ba8fecd40c08323ed8153cfe627c4ea2a9"} Sep 18 06:27:13.331205 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:13.331209 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nzscf" event={"ID":"b642bccf-1547-424f-95ea-6de9749953a1","Type":"ContainerStarted","Data":"cc84d5f2d94b413c9b9dcb14de6655013a1a9fe0a361f5deecf71731c3d98393"} Sep 18 06:27:13.333722 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:13.333694 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-4rqjv" event={"ID":"3b9180cd-aece-4e1a-bd4f-781d7dd5ebf9","Type":"ContainerStarted","Data":"ca89f3f60c6399fafadd31f935718796e898167bc450ec9958595837d51751bd"} Sep 18 06:27:13.333862 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:13.333763 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-4rqjv" event={"ID":"3b9180cd-aece-4e1a-bd4f-781d7dd5ebf9","Type":"ContainerStarted","Data":"bf6c24e49e2ec3e9f099e0d457263156b3422feb49e64f2ce8694f40be11a9aa"} Sep 18 06:27:13.333862 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:13.333794 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-4rqjv" event={"ID":"3b9180cd-aece-4e1a-bd4f-781d7dd5ebf9","Type":"ContainerStarted","Data":"72f7bc7010f7ba61e231ab1aba88b90d13676435ac52d5c3982ae3d749cc4aea"} Sep 18 06:27:13.335713 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:13.335686 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-24gd5" event={"ID":"3fb98b73-fc76-4fab-928c-8c60bb29354c","Type":"ContainerStarted","Data":"d595857d476614bb48fd8d24984077283244ff1f8e4be081fb797431edcec3cd"} Sep 18 06:27:13.335818 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:13.335742 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-24gd5" event={"ID":"3fb98b73-fc76-4fab-928c-8c60bb29354c","Type":"ContainerStarted","Data":"e3d8faf6f271b4d2416276e3bd6f336ae61af4d13be91aae1e1b438b4378ce0d"} Sep 18 06:27:13.335818 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:13.335770 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-24gd5" event={"ID":"3fb98b73-fc76-4fab-928c-8c60bb29354c","Type":"ContainerStarted","Data":"deccb18b26f9a23bf2a46fce454abec8e795ab798e6d707651eaad0defa34d47"} Sep 18 06:27:13.351900 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:13.351835 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-nzscf" podStartSLOduration=3.197035457 podStartE2EDuration="4.351815369s" totalImagesPullingTime="1.085217526s" totalInitContainerRuntime="69.562386ms" isStatefulPod=true podCreationTimestamp="2026-09-18 06:27:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:27:13.35116016 +0000 UTC m=+295.090576363" watchObservedRunningTime="2026-09-18 06:27:13.351815369 +0000 UTC m=+295.091231535" Sep 18 06:27:13.371761 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:13.371703 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-84c885658c-4rqjv" podStartSLOduration=3.017965869 podStartE2EDuration="4.371683289s" totalImagesPullingTime="1.35371742s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:27:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:27:13.371035806 +0000 UTC m=+295.110452002" watchObservedRunningTime="2026-09-18 06:27:13.371683289 +0000 UTC m=+295.111099462" Sep 18 06:27:13.461609 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:13.461525 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-67d649cbb7-jbfsd"] Sep 18 06:27:13.673220 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:27:13.673186 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5c0016b_5ad1_46f0_9460_ae6207e9d171.slice/crio-a18ed71d9c2b95e6068e20c5d528bae9b206f75923b9db3f2c6814be88f0c5bd WatchSource:0}: Error finding container a18ed71d9c2b95e6068e20c5d528bae9b206f75923b9db3f2c6814be88f0c5bd: Status 404 returned error can't find the container with id a18ed71d9c2b95e6068e20c5d528bae9b206f75923b9db3f2c6814be88f0c5bd Sep 18 06:27:14.240709 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:14.240675 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6df5dcb58c-gdgth"] Sep 18 06:27:14.245710 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:14.245069 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6df5dcb58c-gdgth" Sep 18 06:27:14.247506 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:14.247481 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 18 06:27:14.247781 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:14.247766 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 18 06:27:14.248817 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:14.248798 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-p5p8g\"" Sep 18 06:27:14.249110 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:14.249092 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 18 06:27:14.249181 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:14.249117 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-cicd7eedsr21\"" Sep 18 06:27:14.249181 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:14.249103 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 18 06:27:14.253787 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:14.253769 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6df5dcb58c-gdgth"] Sep 18 06:27:14.320871 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:14.320850 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/30956168-2c46-4c18-bd2a-d341ca614112-metrics-server-audit-profiles\") pod \"metrics-server-6df5dcb58c-gdgth\" (UID: \"30956168-2c46-4c18-bd2a-d341ca614112\") " pod="openshift-monitoring/metrics-server-6df5dcb58c-gdgth" Sep 18 06:27:14.320871 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:14.320876 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rkgc\" (UniqueName: \"kubernetes.io/projected/30956168-2c46-4c18-bd2a-d341ca614112-kube-api-access-9rkgc\") pod \"metrics-server-6df5dcb58c-gdgth\" (UID: \"30956168-2c46-4c18-bd2a-d341ca614112\") " pod="openshift-monitoring/metrics-server-6df5dcb58c-gdgth" Sep 18 06:27:14.321026 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:14.320899 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/30956168-2c46-4c18-bd2a-d341ca614112-secret-metrics-server-tls\") pod \"metrics-server-6df5dcb58c-gdgth\" (UID: \"30956168-2c46-4c18-bd2a-d341ca614112\") " pod="openshift-monitoring/metrics-server-6df5dcb58c-gdgth" Sep 18 06:27:14.321026 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:14.320916 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/30956168-2c46-4c18-bd2a-d341ca614112-secret-metrics-server-client-certs\") pod \"metrics-server-6df5dcb58c-gdgth\" (UID: \"30956168-2c46-4c18-bd2a-d341ca614112\") " pod="openshift-monitoring/metrics-server-6df5dcb58c-gdgth" Sep 18 06:27:14.321026 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:14.320981 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30956168-2c46-4c18-bd2a-d341ca614112-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6df5dcb58c-gdgth\" (UID: \"30956168-2c46-4c18-bd2a-d341ca614112\") " pod="openshift-monitoring/metrics-server-6df5dcb58c-gdgth" Sep 18 06:27:14.321177 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:14.321072 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/30956168-2c46-4c18-bd2a-d341ca614112-client-ca-bundle\") pod \"metrics-server-6df5dcb58c-gdgth\" (UID: \"30956168-2c46-4c18-bd2a-d341ca614112\") " pod="openshift-monitoring/metrics-server-6df5dcb58c-gdgth" Sep 18 06:27:14.321177 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:14.321107 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/30956168-2c46-4c18-bd2a-d341ca614112-audit-log\") pod \"metrics-server-6df5dcb58c-gdgth\" (UID: \"30956168-2c46-4c18-bd2a-d341ca614112\") " pod="openshift-monitoring/metrics-server-6df5dcb58c-gdgth" Sep 18 06:27:14.340656 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:14.340611 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67d649cbb7-jbfsd" event={"ID":"d5c0016b-5ad1-46f0-9460-ae6207e9d171","Type":"ContainerStarted","Data":"a18ed71d9c2b95e6068e20c5d528bae9b206f75923b9db3f2c6814be88f0c5bd"} Sep 18 06:27:14.343098 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:14.342998 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-24gd5" event={"ID":"3fb98b73-fc76-4fab-928c-8c60bb29354c","Type":"ContainerStarted","Data":"d41508303a0b6e5d4fd41aa4c837962528c0d67da9a72e32388c38d859cdc901"} Sep 18 06:27:14.360114 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:14.360069 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-24gd5" podStartSLOduration=4.193814266 podStartE2EDuration="5.360055557s" totalImagesPullingTime="1.166241291s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:27:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:27:14.359516399 +0000 UTC m=+296.098932595" watchObservedRunningTime="2026-09-18 06:27:14.360055557 +0000 UTC m=+296.099471731" Sep 18 06:27:14.422153 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:14.422126 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/30956168-2c46-4c18-bd2a-d341ca614112-metrics-server-audit-profiles\") pod \"metrics-server-6df5dcb58c-gdgth\" (UID: \"30956168-2c46-4c18-bd2a-d341ca614112\") " pod="openshift-monitoring/metrics-server-6df5dcb58c-gdgth" Sep 18 06:27:14.422311 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:14.422163 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rkgc\" (UniqueName: \"kubernetes.io/projected/30956168-2c46-4c18-bd2a-d341ca614112-kube-api-access-9rkgc\") pod \"metrics-server-6df5dcb58c-gdgth\" (UID: \"30956168-2c46-4c18-bd2a-d341ca614112\") " pod="openshift-monitoring/metrics-server-6df5dcb58c-gdgth" Sep 18 06:27:14.422311 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:14.422239 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/30956168-2c46-4c18-bd2a-d341ca614112-secret-metrics-server-tls\") pod \"metrics-server-6df5dcb58c-gdgth\" (UID: \"30956168-2c46-4c18-bd2a-d341ca614112\") " pod="openshift-monitoring/metrics-server-6df5dcb58c-gdgth" Sep 18 06:27:14.422311 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:14.422281 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/30956168-2c46-4c18-bd2a-d341ca614112-secret-metrics-server-client-certs\") pod \"metrics-server-6df5dcb58c-gdgth\" (UID: \"30956168-2c46-4c18-bd2a-d341ca614112\") " pod="openshift-monitoring/metrics-server-6df5dcb58c-gdgth" Sep 18 06:27:14.423051 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:14.422620 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30956168-2c46-4c18-bd2a-d341ca614112-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6df5dcb58c-gdgth\" (UID: \"30956168-2c46-4c18-bd2a-d341ca614112\") " pod="openshift-monitoring/metrics-server-6df5dcb58c-gdgth" Sep 18 06:27:14.423051 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:14.422735 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/30956168-2c46-4c18-bd2a-d341ca614112-client-ca-bundle\") pod \"metrics-server-6df5dcb58c-gdgth\" (UID: \"30956168-2c46-4c18-bd2a-d341ca614112\") " pod="openshift-monitoring/metrics-server-6df5dcb58c-gdgth" Sep 18 06:27:14.423051 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:14.422773 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/30956168-2c46-4c18-bd2a-d341ca614112-audit-log\") pod \"metrics-server-6df5dcb58c-gdgth\" (UID: \"30956168-2c46-4c18-bd2a-d341ca614112\") " pod="openshift-monitoring/metrics-server-6df5dcb58c-gdgth" Sep 18 06:27:14.423566 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:14.423285 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30956168-2c46-4c18-bd2a-d341ca614112-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6df5dcb58c-gdgth\" (UID: \"30956168-2c46-4c18-bd2a-d341ca614112\") " pod="openshift-monitoring/metrics-server-6df5dcb58c-gdgth" Sep 18 06:27:14.423566 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:14.423387 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/30956168-2c46-4c18-bd2a-d341ca614112-audit-log\") pod \"metrics-server-6df5dcb58c-gdgth\" (UID: \"30956168-2c46-4c18-bd2a-d341ca614112\") " pod="openshift-monitoring/metrics-server-6df5dcb58c-gdgth" Sep 18 06:27:14.423758 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:14.423689 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/30956168-2c46-4c18-bd2a-d341ca614112-metrics-server-audit-profiles\") pod \"metrics-server-6df5dcb58c-gdgth\" (UID: \"30956168-2c46-4c18-bd2a-d341ca614112\") " pod="openshift-monitoring/metrics-server-6df5dcb58c-gdgth" Sep 18 06:27:14.425441 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:14.425396 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/30956168-2c46-4c18-bd2a-d341ca614112-secret-metrics-server-tls\") pod \"metrics-server-6df5dcb58c-gdgth\" (UID: \"30956168-2c46-4c18-bd2a-d341ca614112\") " pod="openshift-monitoring/metrics-server-6df5dcb58c-gdgth" Sep 18 06:27:14.426068 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:14.426049 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/30956168-2c46-4c18-bd2a-d341ca614112-client-ca-bundle\") pod \"metrics-server-6df5dcb58c-gdgth\" (UID: \"30956168-2c46-4c18-bd2a-d341ca614112\") " pod="openshift-monitoring/metrics-server-6df5dcb58c-gdgth" Sep 18 06:27:14.426278 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:14.426259 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/30956168-2c46-4c18-bd2a-d341ca614112-secret-metrics-server-client-certs\") pod \"metrics-server-6df5dcb58c-gdgth\" (UID: \"30956168-2c46-4c18-bd2a-d341ca614112\") " pod="openshift-monitoring/metrics-server-6df5dcb58c-gdgth" Sep 18 06:27:14.430253 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:14.430230 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rkgc\" (UniqueName: \"kubernetes.io/projected/30956168-2c46-4c18-bd2a-d341ca614112-kube-api-access-9rkgc\") pod \"metrics-server-6df5dcb58c-gdgth\" (UID: \"30956168-2c46-4c18-bd2a-d341ca614112\") " pod="openshift-monitoring/metrics-server-6df5dcb58c-gdgth" Sep 18 06:27:14.440086 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:27:14.440052 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-k77fk" podUID="66dca255-1289-49ec-b200-e950a30e3d9c" Sep 18 06:27:14.555289 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:14.555211 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6df5dcb58c-gdgth" Sep 18 06:27:14.670918 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:14.670889 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-tb24m"] Sep 18 06:27:14.675494 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:14.675465 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-tb24m" Sep 18 06:27:14.678099 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:14.678049 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-vk48f\"" Sep 18 06:27:14.678206 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:14.678049 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 18 06:27:14.683976 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:14.683958 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-tb24m"] Sep 18 06:27:14.726344 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:14.726311 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eff3e28f-3d72-42c6-a14c-3080b1dacf59-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-tb24m\" (UID: \"eff3e28f-3d72-42c6-a14c-3080b1dacf59\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-tb24m" Sep 18 06:27:14.726592 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:14.726430 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qszvh\" (UniqueName: \"kubernetes.io/projected/eff3e28f-3d72-42c6-a14c-3080b1dacf59-kube-api-access-qszvh\") pod \"monitoring-plugin-5984459ff8-tb24m\" (UID: \"eff3e28f-3d72-42c6-a14c-3080b1dacf59\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-tb24m" Sep 18 06:27:14.827659 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:14.827585 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eff3e28f-3d72-42c6-a14c-3080b1dacf59-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-tb24m\" (UID: \"eff3e28f-3d72-42c6-a14c-3080b1dacf59\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-tb24m" Sep 18 06:27:14.827801 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:14.827681 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qszvh\" (UniqueName: \"kubernetes.io/projected/eff3e28f-3d72-42c6-a14c-3080b1dacf59-kube-api-access-qszvh\") pod \"monitoring-plugin-5984459ff8-tb24m\" (UID: \"eff3e28f-3d72-42c6-a14c-3080b1dacf59\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-tb24m" Sep 18 06:27:14.827801 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:27:14.827742 2567 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Sep 18 06:27:14.827884 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:27:14.827821 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eff3e28f-3d72-42c6-a14c-3080b1dacf59-monitoring-plugin-cert podName:eff3e28f-3d72-42c6-a14c-3080b1dacf59 nodeName:}" failed. No retries permitted until 2026-09-18 06:27:15.327800574 +0000 UTC m=+297.067216741 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/eff3e28f-3d72-42c6-a14c-3080b1dacf59-monitoring-plugin-cert") pod "monitoring-plugin-5984459ff8-tb24m" (UID: "eff3e28f-3d72-42c6-a14c-3080b1dacf59") : secret "monitoring-plugin-cert" not found Sep 18 06:27:14.834834 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:14.834809 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6df5dcb58c-gdgth"] Sep 18 06:27:14.838728 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:27:14.838703 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30956168_2c46_4c18_bd2a_d341ca614112.slice/crio-e90f2ff771773dcf66cfcc1c19a793f338bb84d7cb3faf9ebd2545c2c1ff7629 WatchSource:0}: Error finding container e90f2ff771773dcf66cfcc1c19a793f338bb84d7cb3faf9ebd2545c2c1ff7629: Status 404 returned error can't find the container with id e90f2ff771773dcf66cfcc1c19a793f338bb84d7cb3faf9ebd2545c2c1ff7629 Sep 18 06:27:14.839777 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:14.839758 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qszvh\" (UniqueName: \"kubernetes.io/projected/eff3e28f-3d72-42c6-a14c-3080b1dacf59-kube-api-access-qszvh\") pod \"monitoring-plugin-5984459ff8-tb24m\" (UID: \"eff3e28f-3d72-42c6-a14c-3080b1dacf59\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-tb24m" Sep 18 06:27:15.333127 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:15.333091 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eff3e28f-3d72-42c6-a14c-3080b1dacf59-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-tb24m\" (UID: \"eff3e28f-3d72-42c6-a14c-3080b1dacf59\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-tb24m" Sep 18 06:27:15.335917 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:15.335890 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eff3e28f-3d72-42c6-a14c-3080b1dacf59-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-tb24m\" (UID: \"eff3e28f-3d72-42c6-a14c-3080b1dacf59\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-tb24m" Sep 18 06:27:15.348702 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:15.348669 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6df5dcb58c-gdgth" event={"ID":"30956168-2c46-4c18-bd2a-d341ca614112","Type":"ContainerStarted","Data":"e90f2ff771773dcf66cfcc1c19a793f338bb84d7cb3faf9ebd2545c2c1ff7629"} Sep 18 06:27:15.588100 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:15.587098 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-tb24m" Sep 18 06:27:15.887064 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:15.886998 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-tb24m"] Sep 18 06:27:15.889743 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:27:15.889713 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeff3e28f_3d72_42c6_a14c_3080b1dacf59.slice/crio-8704196e39db9c22c5966d9846403424c42f0974b6a4f5fb13ed29cbec6885d7 WatchSource:0}: Error finding container 8704196e39db9c22c5966d9846403424c42f0974b6a4f5fb13ed29cbec6885d7: Status 404 returned error can't find the container with id 8704196e39db9c22c5966d9846403424c42f0974b6a4f5fb13ed29cbec6885d7 Sep 18 06:27:16.355513 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:16.355478 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67d649cbb7-jbfsd" event={"ID":"d5c0016b-5ad1-46f0-9460-ae6207e9d171","Type":"ContainerStarted","Data":"c488946ce05e0112221429c6115af39d4e3e7a1a473e2215000c22cab30a70af"} Sep 18 06:27:16.355946 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:16.355565 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67d649cbb7-jbfsd" event={"ID":"d5c0016b-5ad1-46f0-9460-ae6207e9d171","Type":"ContainerStarted","Data":"a8a93d181c1ab348fdc2bdca73da00e7c286218f7d41383c9c79a2a841844491"} Sep 18 06:27:16.355946 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:16.355606 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67d649cbb7-jbfsd" event={"ID":"d5c0016b-5ad1-46f0-9460-ae6207e9d171","Type":"ContainerStarted","Data":"ffa73d2b38525d9b4743a7e663095b20fa76fb7e1b675ba6748581cea917f424"} Sep 18 06:27:16.356811 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:16.356788 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-tb24m" event={"ID":"eff3e28f-3d72-42c6-a14c-3080b1dacf59","Type":"ContainerStarted","Data":"8704196e39db9c22c5966d9846403424c42f0974b6a4f5fb13ed29cbec6885d7"} Sep 18 06:27:17.308585 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:17.308495 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-ph66s" Sep 18 06:27:17.365416 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:17.365378 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6df5dcb58c-gdgth" event={"ID":"30956168-2c46-4c18-bd2a-d341ca614112","Type":"ContainerStarted","Data":"06bcd14872ac00019745e8531f1634b6514241a26dc861446b8a3b9ebb76ebae"} Sep 18 06:27:17.369711 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:17.369679 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67d649cbb7-jbfsd" event={"ID":"d5c0016b-5ad1-46f0-9460-ae6207e9d171","Type":"ContainerStarted","Data":"8f4d31f9b7291173eee89482c2ace5ea678d6cf26ca71095a31b4f2a2c97eef2"} Sep 18 06:27:17.369850 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:17.369754 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67d649cbb7-jbfsd" event={"ID":"d5c0016b-5ad1-46f0-9460-ae6207e9d171","Type":"ContainerStarted","Data":"c20d46cf15681d1d84e0f9924c43fa02040e45f0bedbfbb71ea2a73439c92de9"} Sep 18 06:27:17.369850 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:17.369798 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67d649cbb7-jbfsd" event={"ID":"d5c0016b-5ad1-46f0-9460-ae6207e9d171","Type":"ContainerStarted","Data":"5dbe2ed75f0c31db686569b4546229a223be8bfcee9ae28c0f9c176d688f3761"} Sep 18 06:27:17.369850 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:17.369844 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-67d649cbb7-jbfsd" Sep 18 06:27:17.382908 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:17.382862 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6df5dcb58c-gdgth" podStartSLOduration=1.597814648 podStartE2EDuration="3.382842616s" totalImagesPullingTime="1.785027968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:27:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:27:17.381152996 +0000 UTC m=+299.120569222" watchObservedRunningTime="2026-09-18 06:27:17.382842616 +0000 UTC m=+299.122258792" Sep 18 06:27:17.401559 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:17.401493 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-67d649cbb7-jbfsd" podStartSLOduration=2.302204867 podStartE2EDuration="5.401476273s" totalImagesPullingTime="3.099271406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:27:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:27:17.400091853 +0000 UTC m=+299.139508056" watchObservedRunningTime="2026-09-18 06:27:17.401476273 +0000 UTC m=+299.140892448" Sep 18 06:27:18.375444 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:18.375410 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-tb24m" event={"ID":"eff3e28f-3d72-42c6-a14c-3080b1dacf59","Type":"ContainerStarted","Data":"eddef845096345df2c0a47f09319aa1d1171fce4272e77cc070d499176e46adc"} Sep 18 06:27:18.375853 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:18.375704 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-tb24m" Sep 18 06:27:18.380169 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:18.380148 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-tb24m" Sep 18 06:27:18.390764 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:18.390727 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5984459ff8-tb24m" podStartSLOduration=2.728342071 podStartE2EDuration="4.390712224s" totalImagesPullingTime="1.662370153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:27:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:27:18.39028205 +0000 UTC m=+300.129698246" watchObservedRunningTime="2026-09-18 06:27:18.390712224 +0000 UTC m=+300.130128397" Sep 18 06:27:20.317662 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:20.317633 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjr4h_eb059951-e263-40da-9155-4fd0cf242820/ovn-acl-logging/0.log" Sep 18 06:27:20.318517 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:20.318502 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjr4h_eb059951-e263-40da-9155-4fd0cf242820/ovn-acl-logging/0.log" Sep 18 06:27:20.324436 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:20.324418 2567 kubelet.go:1648] "Image garbage collection succeeded" Sep 18 06:27:21.431729 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:21.431674 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7cfd7d874-4m95j" podUID="ca057b3a-aa6d-45ff-bad8-6dc74bf2009b" containerName="registry" containerID="cri-o://cc55630ff59a7966111397073c821db036c6946e759d304f6b222e132cea926f" gracePeriod=30 Sep 18 06:27:21.666883 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:21.666857 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cfd7d874-4m95j" Sep 18 06:27:21.806050 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:21.805981 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b-registry-certificates\") pod \"ca057b3a-aa6d-45ff-bad8-6dc74bf2009b\" (UID: \"ca057b3a-aa6d-45ff-bad8-6dc74bf2009b\") " Sep 18 06:27:21.806050 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:21.806019 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b-trusted-ca\") pod \"ca057b3a-aa6d-45ff-bad8-6dc74bf2009b\" (UID: \"ca057b3a-aa6d-45ff-bad8-6dc74bf2009b\") " Sep 18 06:27:21.806050 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:21.806038 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b-bound-sa-token\") pod \"ca057b3a-aa6d-45ff-bad8-6dc74bf2009b\" (UID: \"ca057b3a-aa6d-45ff-bad8-6dc74bf2009b\") " Sep 18 06:27:21.806273 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:21.806071 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b-installation-pull-secrets\") pod \"ca057b3a-aa6d-45ff-bad8-6dc74bf2009b\" (UID: \"ca057b3a-aa6d-45ff-bad8-6dc74bf2009b\") " Sep 18 06:27:21.806273 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:21.806090 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b-ca-trust-extracted\") pod \"ca057b3a-aa6d-45ff-bad8-6dc74bf2009b\" (UID: \"ca057b3a-aa6d-45ff-bad8-6dc74bf2009b\") " Sep 18 06:27:21.806273 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:21.806125 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b-image-registry-private-configuration\") pod \"ca057b3a-aa6d-45ff-bad8-6dc74bf2009b\" (UID: \"ca057b3a-aa6d-45ff-bad8-6dc74bf2009b\") " Sep 18 06:27:21.806273 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:21.806152 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-csm7x\" (UniqueName: \"kubernetes.io/projected/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b-kube-api-access-csm7x\") pod \"ca057b3a-aa6d-45ff-bad8-6dc74bf2009b\" (UID: \"ca057b3a-aa6d-45ff-bad8-6dc74bf2009b\") " Sep 18 06:27:21.806273 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:21.806179 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b-registry-tls\") pod \"ca057b3a-aa6d-45ff-bad8-6dc74bf2009b\" (UID: \"ca057b3a-aa6d-45ff-bad8-6dc74bf2009b\") " Sep 18 06:27:21.806555 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:21.806488 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "ca057b3a-aa6d-45ff-bad8-6dc74bf2009b" (UID: "ca057b3a-aa6d-45ff-bad8-6dc74bf2009b"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 06:27:21.806712 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:21.806673 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "ca057b3a-aa6d-45ff-bad8-6dc74bf2009b" (UID: "ca057b3a-aa6d-45ff-bad8-6dc74bf2009b"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 06:27:21.808584 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:21.808540 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "ca057b3a-aa6d-45ff-bad8-6dc74bf2009b" (UID: "ca057b3a-aa6d-45ff-bad8-6dc74bf2009b"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 06:27:21.808584 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:21.808559 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b-kube-api-access-csm7x" (OuterVolumeSpecName: "kube-api-access-csm7x") pod "ca057b3a-aa6d-45ff-bad8-6dc74bf2009b" (UID: "ca057b3a-aa6d-45ff-bad8-6dc74bf2009b"). InnerVolumeSpecName "kube-api-access-csm7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 06:27:21.808742 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:21.808632 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "ca057b3a-aa6d-45ff-bad8-6dc74bf2009b" (UID: "ca057b3a-aa6d-45ff-bad8-6dc74bf2009b"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 06:27:21.808742 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:21.808627 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "ca057b3a-aa6d-45ff-bad8-6dc74bf2009b" (UID: "ca057b3a-aa6d-45ff-bad8-6dc74bf2009b"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 06:27:21.808893 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:21.808876 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "ca057b3a-aa6d-45ff-bad8-6dc74bf2009b" (UID: "ca057b3a-aa6d-45ff-bad8-6dc74bf2009b"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 06:27:21.814546 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:21.814526 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "ca057b3a-aa6d-45ff-bad8-6dc74bf2009b" (UID: "ca057b3a-aa6d-45ff-bad8-6dc74bf2009b"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:27:21.907378 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:21.907347 2567 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b-installation-pull-secrets\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:27:21.907456 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:21.907386 2567 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b-ca-trust-extracted\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:27:21.907456 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:21.907397 2567 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b-image-registry-private-configuration\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:27:21.907456 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:21.907406 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csm7x\" (UniqueName: \"kubernetes.io/projected/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b-kube-api-access-csm7x\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:27:21.907456 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:21.907415 2567 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b-registry-tls\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:27:21.907456 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:21.907424 2567 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b-registry-certificates\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:27:21.907456 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:21.907432 2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b-trusted-ca\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:27:21.907456 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:21.907440 2567 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b-bound-sa-token\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:27:22.391417 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:22.391386 2567 generic.go:356] "Generic (PLEG): container finished" podID="ca057b3a-aa6d-45ff-bad8-6dc74bf2009b" containerID="cc55630ff59a7966111397073c821db036c6946e759d304f6b222e132cea926f" exitCode=0 Sep 18 06:27:22.391567 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:22.391429 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cfd7d874-4m95j" Sep 18 06:27:22.391567 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:22.391454 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cfd7d874-4m95j" event={"ID":"ca057b3a-aa6d-45ff-bad8-6dc74bf2009b","Type":"ContainerDied","Data":"cc55630ff59a7966111397073c821db036c6946e759d304f6b222e132cea926f"} Sep 18 06:27:22.391567 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:22.391507 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cfd7d874-4m95j" event={"ID":"ca057b3a-aa6d-45ff-bad8-6dc74bf2009b","Type":"ContainerDied","Data":"eb64e68340cdb2baebfdab222ec8465663b3e81f30f563ea4e7cc24e5aa6f500"} Sep 18 06:27:22.391567 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:22.391535 2567 scope.go:117] "RemoveContainer" containerID="cc55630ff59a7966111397073c821db036c6946e759d304f6b222e132cea926f" Sep 18 06:27:22.406471 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:22.406444 2567 scope.go:117] "RemoveContainer" containerID="cc55630ff59a7966111397073c821db036c6946e759d304f6b222e132cea926f" Sep 18 06:27:22.406889 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:27:22.406868 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc55630ff59a7966111397073c821db036c6946e759d304f6b222e132cea926f\": container with ID starting with cc55630ff59a7966111397073c821db036c6946e759d304f6b222e132cea926f not found: ID does not exist" containerID="cc55630ff59a7966111397073c821db036c6946e759d304f6b222e132cea926f" Sep 18 06:27:22.406965 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:22.406896 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc55630ff59a7966111397073c821db036c6946e759d304f6b222e132cea926f"} err="failed to get container status \"cc55630ff59a7966111397073c821db036c6946e759d304f6b222e132cea926f\": rpc error: code = NotFound desc = could not find container \"cc55630ff59a7966111397073c821db036c6946e759d304f6b222e132cea926f\": container with ID starting with cc55630ff59a7966111397073c821db036c6946e759d304f6b222e132cea926f not found: ID does not exist" Sep 18 06:27:22.415725 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:22.415704 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7cfd7d874-4m95j"] Sep 18 06:27:22.419061 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:22.419041 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7cfd7d874-4m95j"] Sep 18 06:27:22.444543 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:22.444520 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca057b3a-aa6d-45ff-bad8-6dc74bf2009b" path="/var/lib/kubelet/pods/ca057b3a-aa6d-45ff-bad8-6dc74bf2009b/volumes" Sep 18 06:27:23.381096 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:23.381069 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-67d649cbb7-jbfsd" Sep 18 06:27:29.439281 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:29.439251 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k77fk" Sep 18 06:27:34.555291 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:34.555248 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6df5dcb58c-gdgth" Sep 18 06:27:34.555684 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:34.555306 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6df5dcb58c-gdgth" Sep 18 06:27:35.874970 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:35.874942 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-84c9ccfcc7-d9gw6"] Sep 18 06:27:35.875243 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:35.875225 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca057b3a-aa6d-45ff-bad8-6dc74bf2009b" containerName="registry" Sep 18 06:27:35.875243 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:35.875237 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca057b3a-aa6d-45ff-bad8-6dc74bf2009b" containerName="registry" Sep 18 06:27:35.875316 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:35.875295 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca057b3a-aa6d-45ff-bad8-6dc74bf2009b" containerName="registry" Sep 18 06:27:35.877898 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:35.877884 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84c9ccfcc7-d9gw6" Sep 18 06:27:35.880248 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:35.880231 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 18 06:27:35.880374 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:35.880346 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 18 06:27:35.881219 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:35.881205 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-q62qv\"" Sep 18 06:27:35.881270 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:35.881204 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 18 06:27:35.881397 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:35.881383 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 18 06:27:35.881435 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:35.881384 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 18 06:27:35.881624 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:35.881606 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 18 06:27:35.881660 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:35.881622 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 18 06:27:35.885807 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:35.885788 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84c9ccfcc7-d9gw6"] Sep 18 06:27:36.015141 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:36.015120 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e74edc21-cd8e-4c0d-a71f-020c6f1e99e0-console-config\") pod \"console-84c9ccfcc7-d9gw6\" (UID: \"e74edc21-cd8e-4c0d-a71f-020c6f1e99e0\") " pod="openshift-console/console-84c9ccfcc7-d9gw6" Sep 18 06:27:36.015253 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:36.015146 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e74edc21-cd8e-4c0d-a71f-020c6f1e99e0-oauth-serving-cert\") pod \"console-84c9ccfcc7-d9gw6\" (UID: \"e74edc21-cd8e-4c0d-a71f-020c6f1e99e0\") " pod="openshift-console/console-84c9ccfcc7-d9gw6" Sep 18 06:27:36.015253 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:36.015187 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e74edc21-cd8e-4c0d-a71f-020c6f1e99e0-console-serving-cert\") pod \"console-84c9ccfcc7-d9gw6\" (UID: \"e74edc21-cd8e-4c0d-a71f-020c6f1e99e0\") " pod="openshift-console/console-84c9ccfcc7-d9gw6" Sep 18 06:27:36.015253 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:36.015211 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5wx8\" (UniqueName: \"kubernetes.io/projected/e74edc21-cd8e-4c0d-a71f-020c6f1e99e0-kube-api-access-n5wx8\") pod \"console-84c9ccfcc7-d9gw6\" (UID: \"e74edc21-cd8e-4c0d-a71f-020c6f1e99e0\") " pod="openshift-console/console-84c9ccfcc7-d9gw6" Sep 18 06:27:36.015253 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:36.015238 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e74edc21-cd8e-4c0d-a71f-020c6f1e99e0-console-oauth-config\") pod \"console-84c9ccfcc7-d9gw6\" (UID: \"e74edc21-cd8e-4c0d-a71f-020c6f1e99e0\") " pod="openshift-console/console-84c9ccfcc7-d9gw6" Sep 18 06:27:36.015410 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:36.015271 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e74edc21-cd8e-4c0d-a71f-020c6f1e99e0-service-ca\") pod \"console-84c9ccfcc7-d9gw6\" (UID: \"e74edc21-cd8e-4c0d-a71f-020c6f1e99e0\") " pod="openshift-console/console-84c9ccfcc7-d9gw6" Sep 18 06:27:36.116315 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:36.116242 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e74edc21-cd8e-4c0d-a71f-020c6f1e99e0-service-ca\") pod \"console-84c9ccfcc7-d9gw6\" (UID: \"e74edc21-cd8e-4c0d-a71f-020c6f1e99e0\") " pod="openshift-console/console-84c9ccfcc7-d9gw6" Sep 18 06:27:36.116315 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:36.116289 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e74edc21-cd8e-4c0d-a71f-020c6f1e99e0-console-config\") pod \"console-84c9ccfcc7-d9gw6\" (UID: \"e74edc21-cd8e-4c0d-a71f-020c6f1e99e0\") " pod="openshift-console/console-84c9ccfcc7-d9gw6" Sep 18 06:27:36.116315 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:36.116307 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e74edc21-cd8e-4c0d-a71f-020c6f1e99e0-oauth-serving-cert\") pod \"console-84c9ccfcc7-d9gw6\" (UID: \"e74edc21-cd8e-4c0d-a71f-020c6f1e99e0\") " pod="openshift-console/console-84c9ccfcc7-d9gw6" Sep 18 06:27:36.116626 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:36.116348 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e74edc21-cd8e-4c0d-a71f-020c6f1e99e0-console-serving-cert\") pod \"console-84c9ccfcc7-d9gw6\" (UID: \"e74edc21-cd8e-4c0d-a71f-020c6f1e99e0\") " pod="openshift-console/console-84c9ccfcc7-d9gw6" Sep 18 06:27:36.116626 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:36.116386 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5wx8\" (UniqueName: \"kubernetes.io/projected/e74edc21-cd8e-4c0d-a71f-020c6f1e99e0-kube-api-access-n5wx8\") pod \"console-84c9ccfcc7-d9gw6\" (UID: \"e74edc21-cd8e-4c0d-a71f-020c6f1e99e0\") " pod="openshift-console/console-84c9ccfcc7-d9gw6" Sep 18 06:27:36.116626 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:36.116415 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e74edc21-cd8e-4c0d-a71f-020c6f1e99e0-console-oauth-config\") pod \"console-84c9ccfcc7-d9gw6\" (UID: \"e74edc21-cd8e-4c0d-a71f-020c6f1e99e0\") " pod="openshift-console/console-84c9ccfcc7-d9gw6" Sep 18 06:27:36.117151 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:36.117123 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e74edc21-cd8e-4c0d-a71f-020c6f1e99e0-console-config\") pod \"console-84c9ccfcc7-d9gw6\" (UID: \"e74edc21-cd8e-4c0d-a71f-020c6f1e99e0\") " pod="openshift-console/console-84c9ccfcc7-d9gw6" Sep 18 06:27:36.117245 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:36.117155 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e74edc21-cd8e-4c0d-a71f-020c6f1e99e0-service-ca\") pod \"console-84c9ccfcc7-d9gw6\" (UID: \"e74edc21-cd8e-4c0d-a71f-020c6f1e99e0\") " pod="openshift-console/console-84c9ccfcc7-d9gw6" Sep 18 06:27:36.117474 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:36.117449 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e74edc21-cd8e-4c0d-a71f-020c6f1e99e0-oauth-serving-cert\") pod \"console-84c9ccfcc7-d9gw6\" (UID: \"e74edc21-cd8e-4c0d-a71f-020c6f1e99e0\") " pod="openshift-console/console-84c9ccfcc7-d9gw6" Sep 18 06:27:36.118952 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:36.118931 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e74edc21-cd8e-4c0d-a71f-020c6f1e99e0-console-serving-cert\") pod \"console-84c9ccfcc7-d9gw6\" (UID: \"e74edc21-cd8e-4c0d-a71f-020c6f1e99e0\") " pod="openshift-console/console-84c9ccfcc7-d9gw6" Sep 18 06:27:36.119068 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:36.119047 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e74edc21-cd8e-4c0d-a71f-020c6f1e99e0-console-oauth-config\") pod \"console-84c9ccfcc7-d9gw6\" (UID: \"e74edc21-cd8e-4c0d-a71f-020c6f1e99e0\") " pod="openshift-console/console-84c9ccfcc7-d9gw6" Sep 18 06:27:36.123708 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:36.123690 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5wx8\" (UniqueName: \"kubernetes.io/projected/e74edc21-cd8e-4c0d-a71f-020c6f1e99e0-kube-api-access-n5wx8\") pod \"console-84c9ccfcc7-d9gw6\" (UID: \"e74edc21-cd8e-4c0d-a71f-020c6f1e99e0\") " pod="openshift-console/console-84c9ccfcc7-d9gw6" Sep 18 06:27:36.224953 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:36.224926 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84c9ccfcc7-d9gw6" Sep 18 06:27:36.442423 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:36.442338 2567 generic.go:356] "Generic (PLEG): container finished" podID="ee4fa404-921d-48d8-a26d-fa5c15425928" containerID="d3014bd3c84bbf53ce29dea05cdbd75bae13df6a5461088e2923113348873169" exitCode=0 Sep 18 06:27:36.444424 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:36.444399 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-5bn97" event={"ID":"ee4fa404-921d-48d8-a26d-fa5c15425928","Type":"ContainerDied","Data":"d3014bd3c84bbf53ce29dea05cdbd75bae13df6a5461088e2923113348873169"} Sep 18 06:27:36.444667 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:36.444652 2567 scope.go:117] "RemoveContainer" containerID="d3014bd3c84bbf53ce29dea05cdbd75bae13df6a5461088e2923113348873169" Sep 18 06:27:36.494102 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:36.494080 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84c9ccfcc7-d9gw6"] Sep 18 06:27:36.497935 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:27:36.497905 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode74edc21_cd8e_4c0d_a71f_020c6f1e99e0.slice/crio-3ace54a30778a9c5631a04d7c36b7f415ea318c8237e2fb3657abfa3581be46b WatchSource:0}: Error finding container 3ace54a30778a9c5631a04d7c36b7f415ea318c8237e2fb3657abfa3581be46b: Status 404 returned error can't find the container with id 3ace54a30778a9c5631a04d7c36b7f415ea318c8237e2fb3657abfa3581be46b Sep 18 06:27:36.500252 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:36.500229 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 18 06:27:36.960075 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:36.960031 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-75b6d7b44b-fvj8r_e16d183f-ea45-4a4c-94be-f078321c3c79/router/0.log" Sep 18 06:27:36.966933 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:36.966908 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qxhxp_3320df23-25b7-4e16-9024-076401f7ca3e/serve-healthcheck-canary/0.log" Sep 18 06:27:37.458220 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:37.458180 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-5bn97" event={"ID":"ee4fa404-921d-48d8-a26d-fa5c15425928","Type":"ContainerStarted","Data":"dcce7b486bb59ffda3726a64127a92c6b04868294669a43966ca82b0142fc293"} Sep 18 06:27:37.460527 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:37.460465 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84c9ccfcc7-d9gw6" event={"ID":"e74edc21-cd8e-4c0d-a71f-020c6f1e99e0","Type":"ContainerStarted","Data":"3ace54a30778a9c5631a04d7c36b7f415ea318c8237e2fb3657abfa3581be46b"} Sep 18 06:27:40.476737 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:40.476701 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84c9ccfcc7-d9gw6" event={"ID":"e74edc21-cd8e-4c0d-a71f-020c6f1e99e0","Type":"ContainerStarted","Data":"485985efd06b647838fe674086899a719b9b9014ca02e126a47c63e9e29df266"} Sep 18 06:27:40.496791 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:40.496746 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-84c9ccfcc7-d9gw6" podStartSLOduration=1.6416636439999999 podStartE2EDuration="5.496731492s" totalImagesPullingTime="3.855067848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:27:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:27:40.494678128 +0000 UTC m=+322.234094306" watchObservedRunningTime="2026-09-18 06:27:40.496731492 +0000 UTC m=+322.236147669" Sep 18 06:27:46.226014 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:46.225982 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-84c9ccfcc7-d9gw6" Sep 18 06:27:46.226477 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:46.226054 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-84c9ccfcc7-d9gw6" Sep 18 06:27:46.230709 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:46.230689 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-84c9ccfcc7-d9gw6" Sep 18 06:27:46.502407 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:46.502320 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-84c9ccfcc7-d9gw6" Sep 18 06:27:54.561347 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:54.561317 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6df5dcb58c-gdgth" Sep 18 06:27:54.565383 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:54.565338 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6df5dcb58c-gdgth" Sep 18 06:27:56.855630 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:27:56.855596 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84c9ccfcc7-d9gw6"] Sep 18 06:28:01.561279 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:01.561248 2567 generic.go:356] "Generic (PLEG): container finished" podID="b856fd0d-7ea9-4899-958a-98efec61fe44" containerID="e69cfdab794fe6fb2cf15abeaf2753b437a5e214c10fedf8ea96fb9fb3d53783" exitCode=0 Sep 18 06:28:01.561683 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:01.561324 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-kzkk8" event={"ID":"b856fd0d-7ea9-4899-958a-98efec61fe44","Type":"ContainerDied","Data":"e69cfdab794fe6fb2cf15abeaf2753b437a5e214c10fedf8ea96fb9fb3d53783"} Sep 18 06:28:01.561683 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:01.561632 2567 scope.go:117] "RemoveContainer" containerID="e69cfdab794fe6fb2cf15abeaf2753b437a5e214c10fedf8ea96fb9fb3d53783" Sep 18 06:28:02.566955 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:02.566884 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-kzkk8" event={"ID":"b856fd0d-7ea9-4899-958a-98efec61fe44","Type":"ContainerStarted","Data":"f615fa931a6851ab4de869c875dbe9c1fe8e0b581e902f48b72cbd5e1ca92c35"} Sep 18 06:28:21.875671 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:21.875629 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-84c9ccfcc7-d9gw6" podUID="e74edc21-cd8e-4c0d-a71f-020c6f1e99e0" containerName="console" containerID="cri-o://485985efd06b647838fe674086899a719b9b9014ca02e126a47c63e9e29df266" gracePeriod=15 Sep 18 06:28:22.103909 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:22.103886 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84c9ccfcc7-d9gw6" Sep 18 06:28:22.168356 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:22.168293 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e74edc21-cd8e-4c0d-a71f-020c6f1e99e0-console-config\") pod \"e74edc21-cd8e-4c0d-a71f-020c6f1e99e0\" (UID: \"e74edc21-cd8e-4c0d-a71f-020c6f1e99e0\") " Sep 18 06:28:22.168356 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:22.168335 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e74edc21-cd8e-4c0d-a71f-020c6f1e99e0-console-serving-cert\") pod \"e74edc21-cd8e-4c0d-a71f-020c6f1e99e0\" (UID: \"e74edc21-cd8e-4c0d-a71f-020c6f1e99e0\") " Sep 18 06:28:22.168514 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:22.168387 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n5wx8\" (UniqueName: \"kubernetes.io/projected/e74edc21-cd8e-4c0d-a71f-020c6f1e99e0-kube-api-access-n5wx8\") pod \"e74edc21-cd8e-4c0d-a71f-020c6f1e99e0\" (UID: \"e74edc21-cd8e-4c0d-a71f-020c6f1e99e0\") " Sep 18 06:28:22.168514 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:22.168412 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e74edc21-cd8e-4c0d-a71f-020c6f1e99e0-console-oauth-config\") pod \"e74edc21-cd8e-4c0d-a71f-020c6f1e99e0\" (UID: \"e74edc21-cd8e-4c0d-a71f-020c6f1e99e0\") " Sep 18 06:28:22.168514 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:22.168469 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e74edc21-cd8e-4c0d-a71f-020c6f1e99e0-oauth-serving-cert\") pod \"e74edc21-cd8e-4c0d-a71f-020c6f1e99e0\" (UID: \"e74edc21-cd8e-4c0d-a71f-020c6f1e99e0\") " Sep 18 06:28:22.168514 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:22.168491 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e74edc21-cd8e-4c0d-a71f-020c6f1e99e0-service-ca\") pod \"e74edc21-cd8e-4c0d-a71f-020c6f1e99e0\" (UID: \"e74edc21-cd8e-4c0d-a71f-020c6f1e99e0\") " Sep 18 06:28:22.168824 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:22.168718 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e74edc21-cd8e-4c0d-a71f-020c6f1e99e0-console-config" (OuterVolumeSpecName: "console-config") pod "e74edc21-cd8e-4c0d-a71f-020c6f1e99e0" (UID: "e74edc21-cd8e-4c0d-a71f-020c6f1e99e0"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 06:28:22.168900 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:22.168873 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e74edc21-cd8e-4c0d-a71f-020c6f1e99e0-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e74edc21-cd8e-4c0d-a71f-020c6f1e99e0" (UID: "e74edc21-cd8e-4c0d-a71f-020c6f1e99e0"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 06:28:22.168900 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:22.168883 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e74edc21-cd8e-4c0d-a71f-020c6f1e99e0-service-ca" (OuterVolumeSpecName: "service-ca") pod "e74edc21-cd8e-4c0d-a71f-020c6f1e99e0" (UID: "e74edc21-cd8e-4c0d-a71f-020c6f1e99e0"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 06:28:22.170657 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:22.170630 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e74edc21-cd8e-4c0d-a71f-020c6f1e99e0-kube-api-access-n5wx8" (OuterVolumeSpecName: "kube-api-access-n5wx8") pod "e74edc21-cd8e-4c0d-a71f-020c6f1e99e0" (UID: "e74edc21-cd8e-4c0d-a71f-020c6f1e99e0"). InnerVolumeSpecName "kube-api-access-n5wx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 06:28:22.170741 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:22.170681 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e74edc21-cd8e-4c0d-a71f-020c6f1e99e0-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e74edc21-cd8e-4c0d-a71f-020c6f1e99e0" (UID: "e74edc21-cd8e-4c0d-a71f-020c6f1e99e0"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 06:28:22.170780 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:22.170736 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e74edc21-cd8e-4c0d-a71f-020c6f1e99e0-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e74edc21-cd8e-4c0d-a71f-020c6f1e99e0" (UID: "e74edc21-cd8e-4c0d-a71f-020c6f1e99e0"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 06:28:22.269550 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:22.269521 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5wx8\" (UniqueName: \"kubernetes.io/projected/e74edc21-cd8e-4c0d-a71f-020c6f1e99e0-kube-api-access-n5wx8\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:28:22.269550 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:22.269547 2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e74edc21-cd8e-4c0d-a71f-020c6f1e99e0-console-oauth-config\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:28:22.269664 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:22.269557 2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e74edc21-cd8e-4c0d-a71f-020c6f1e99e0-oauth-serving-cert\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:28:22.269664 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:22.269566 2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e74edc21-cd8e-4c0d-a71f-020c6f1e99e0-service-ca\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:28:22.269664 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:22.269575 2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e74edc21-cd8e-4c0d-a71f-020c6f1e99e0-console-config\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:28:22.269664 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:22.269582 2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e74edc21-cd8e-4c0d-a71f-020c6f1e99e0-console-serving-cert\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:28:22.643027 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:22.642997 2567 generic.go:356] "Generic (PLEG): container finished" podID="e74edc21-cd8e-4c0d-a71f-020c6f1e99e0" containerID="485985efd06b647838fe674086899a719b9b9014ca02e126a47c63e9e29df266" exitCode=0 Sep 18 06:28:22.643175 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:22.643052 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84c9ccfcc7-d9gw6" Sep 18 06:28:22.643175 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:22.643081 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84c9ccfcc7-d9gw6" event={"ID":"e74edc21-cd8e-4c0d-a71f-020c6f1e99e0","Type":"ContainerDied","Data":"485985efd06b647838fe674086899a719b9b9014ca02e126a47c63e9e29df266"} Sep 18 06:28:22.643175 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:22.643128 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84c9ccfcc7-d9gw6" event={"ID":"e74edc21-cd8e-4c0d-a71f-020c6f1e99e0","Type":"ContainerDied","Data":"3ace54a30778a9c5631a04d7c36b7f415ea318c8237e2fb3657abfa3581be46b"} Sep 18 06:28:22.643175 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:22.643147 2567 scope.go:117] "RemoveContainer" containerID="485985efd06b647838fe674086899a719b9b9014ca02e126a47c63e9e29df266" Sep 18 06:28:22.671398 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:22.671356 2567 scope.go:117] "RemoveContainer" containerID="485985efd06b647838fe674086899a719b9b9014ca02e126a47c63e9e29df266" Sep 18 06:28:22.671675 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:28:22.671654 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"485985efd06b647838fe674086899a719b9b9014ca02e126a47c63e9e29df266\": container with ID starting with 485985efd06b647838fe674086899a719b9b9014ca02e126a47c63e9e29df266 not found: ID does not exist" containerID="485985efd06b647838fe674086899a719b9b9014ca02e126a47c63e9e29df266" Sep 18 06:28:22.671734 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:22.671684 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"485985efd06b647838fe674086899a719b9b9014ca02e126a47c63e9e29df266"} err="failed to get container status \"485985efd06b647838fe674086899a719b9b9014ca02e126a47c63e9e29df266\": rpc error: code = NotFound desc = could not find container \"485985efd06b647838fe674086899a719b9b9014ca02e126a47c63e9e29df266\": container with ID starting with 485985efd06b647838fe674086899a719b9b9014ca02e126a47c63e9e29df266 not found: ID does not exist" Sep 18 06:28:22.679174 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:22.679152 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84c9ccfcc7-d9gw6"] Sep 18 06:28:22.682255 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:22.682235 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-84c9ccfcc7-d9gw6"] Sep 18 06:28:24.445530 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:24.445490 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e74edc21-cd8e-4c0d-a71f-020c6f1e99e0" path="/var/lib/kubelet/pods/e74edc21-cd8e-4c0d-a71f-020c6f1e99e0/volumes" Sep 18 06:28:34.462996 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:34.462967 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66dca255-1289-49ec-b200-e950a30e3d9c-metrics-certs\") pod \"network-metrics-daemon-k77fk\" (UID: \"66dca255-1289-49ec-b200-e950a30e3d9c\") " pod="openshift-multus/network-metrics-daemon-k77fk" Sep 18 06:28:34.465939 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:34.465910 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66dca255-1289-49ec-b200-e950a30e3d9c-metrics-certs\") pod \"network-metrics-daemon-k77fk\" (UID: \"66dca255-1289-49ec-b200-e950a30e3d9c\") " pod="openshift-multus/network-metrics-daemon-k77fk" Sep 18 06:28:34.542763 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:34.542741 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fg7cb\"" Sep 18 06:28:34.550885 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:34.550857 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k77fk" Sep 18 06:28:34.815975 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:34.815902 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-k77fk"] Sep 18 06:28:34.818719 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:28:34.818692 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66dca255_1289_49ec_b200_e950a30e3d9c.slice/crio-f9e673d4ad012639c4c57e6c02d103bf66bd79aa9bf2ca3e30987fe6d120fe04 WatchSource:0}: Error finding container f9e673d4ad012639c4c57e6c02d103bf66bd79aa9bf2ca3e30987fe6d120fe04: Status 404 returned error can't find the container with id f9e673d4ad012639c4c57e6c02d103bf66bd79aa9bf2ca3e30987fe6d120fe04 Sep 18 06:28:35.694417 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:35.694387 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-k77fk" event={"ID":"66dca255-1289-49ec-b200-e950a30e3d9c","Type":"ContainerStarted","Data":"f9e673d4ad012639c4c57e6c02d103bf66bd79aa9bf2ca3e30987fe6d120fe04"} Sep 18 06:28:36.699848 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:36.699811 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-k77fk" event={"ID":"66dca255-1289-49ec-b200-e950a30e3d9c","Type":"ContainerStarted","Data":"f0e71ac8f713f3bb21b221ca127c6fe5ae350a24c1e5c8b30dd8e2f976b1b07d"} Sep 18 06:28:36.700259 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:36.699904 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-k77fk" event={"ID":"66dca255-1289-49ec-b200-e950a30e3d9c","Type":"ContainerStarted","Data":"3c5b96af17289c8f410f8635b788bb0737f7d8ae30d0a35c4b6a40a95a0f25ae"} Sep 18 06:28:36.717311 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:36.717259 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-k77fk" podStartSLOduration=375.889798687 podStartE2EDuration="6m16.717244153s" totalImagesPullingTime="827.445466ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:22:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:28:36.716202779 +0000 UTC m=+378.455618972" watchObservedRunningTime="2026-09-18 06:28:36.717244153 +0000 UTC m=+378.456660326" Sep 18 06:28:37.074085 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:37.073998 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-ffd5bf477-69lqk"] Sep 18 06:28:37.074317 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:37.074304 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e74edc21-cd8e-4c0d-a71f-020c6f1e99e0" containerName="console" Sep 18 06:28:37.074353 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:37.074318 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e74edc21-cd8e-4c0d-a71f-020c6f1e99e0" containerName="console" Sep 18 06:28:37.074418 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:37.074395 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="e74edc21-cd8e-4c0d-a71f-020c6f1e99e0" containerName="console" Sep 18 06:28:37.076564 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:37.076541 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ffd5bf477-69lqk" Sep 18 06:28:37.078961 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:37.078936 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 18 06:28:37.079071 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:37.078936 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-q62qv\"" Sep 18 06:28:37.079071 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:37.079014 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 18 06:28:37.079283 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:37.079268 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 18 06:28:37.079376 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:37.079302 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 18 06:28:37.079977 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:37.079960 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 18 06:28:37.080057 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:37.079966 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 18 06:28:37.080113 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:37.080098 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 18 06:28:37.083993 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:37.083968 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 18 06:28:37.089700 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:37.089678 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-ffd5bf477-69lqk"] Sep 18 06:28:37.187954 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:37.187916 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgvw2\" (UniqueName: \"kubernetes.io/projected/7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25-kube-api-access-dgvw2\") pod \"console-ffd5bf477-69lqk\" (UID: \"7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25\") " pod="openshift-console/console-ffd5bf477-69lqk" Sep 18 06:28:37.187954 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:37.187961 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25-service-ca\") pod \"console-ffd5bf477-69lqk\" (UID: \"7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25\") " pod="openshift-console/console-ffd5bf477-69lqk" Sep 18 06:28:37.188216 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:37.187988 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25-console-oauth-config\") pod \"console-ffd5bf477-69lqk\" (UID: \"7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25\") " pod="openshift-console/console-ffd5bf477-69lqk" Sep 18 06:28:37.188216 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:37.188015 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25-trusted-ca-bundle\") pod \"console-ffd5bf477-69lqk\" (UID: \"7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25\") " pod="openshift-console/console-ffd5bf477-69lqk" Sep 18 06:28:37.188216 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:37.188038 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25-console-serving-cert\") pod \"console-ffd5bf477-69lqk\" (UID: \"7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25\") " pod="openshift-console/console-ffd5bf477-69lqk" Sep 18 06:28:37.188216 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:37.188143 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25-console-config\") pod \"console-ffd5bf477-69lqk\" (UID: \"7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25\") " pod="openshift-console/console-ffd5bf477-69lqk" Sep 18 06:28:37.188216 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:37.188191 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25-oauth-serving-cert\") pod \"console-ffd5bf477-69lqk\" (UID: \"7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25\") " pod="openshift-console/console-ffd5bf477-69lqk" Sep 18 06:28:37.289451 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:37.289344 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25-console-config\") pod \"console-ffd5bf477-69lqk\" (UID: \"7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25\") " pod="openshift-console/console-ffd5bf477-69lqk" Sep 18 06:28:37.289451 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:37.289419 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25-oauth-serving-cert\") pod \"console-ffd5bf477-69lqk\" (UID: \"7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25\") " pod="openshift-console/console-ffd5bf477-69lqk" Sep 18 06:28:37.289451 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:37.289456 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgvw2\" (UniqueName: \"kubernetes.io/projected/7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25-kube-api-access-dgvw2\") pod \"console-ffd5bf477-69lqk\" (UID: \"7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25\") " pod="openshift-console/console-ffd5bf477-69lqk" Sep 18 06:28:37.289766 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:37.289489 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25-service-ca\") pod \"console-ffd5bf477-69lqk\" (UID: \"7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25\") " pod="openshift-console/console-ffd5bf477-69lqk" Sep 18 06:28:37.289766 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:37.289626 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25-console-oauth-config\") pod \"console-ffd5bf477-69lqk\" (UID: \"7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25\") " pod="openshift-console/console-ffd5bf477-69lqk" Sep 18 06:28:37.289766 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:37.289677 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25-trusted-ca-bundle\") pod \"console-ffd5bf477-69lqk\" (UID: \"7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25\") " pod="openshift-console/console-ffd5bf477-69lqk" Sep 18 06:28:37.289766 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:37.289711 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25-console-serving-cert\") pod \"console-ffd5bf477-69lqk\" (UID: \"7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25\") " pod="openshift-console/console-ffd5bf477-69lqk" Sep 18 06:28:37.290212 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:37.290188 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25-service-ca\") pod \"console-ffd5bf477-69lqk\" (UID: \"7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25\") " pod="openshift-console/console-ffd5bf477-69lqk" Sep 18 06:28:37.290307 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:37.290242 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25-oauth-serving-cert\") pod \"console-ffd5bf477-69lqk\" (UID: \"7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25\") " pod="openshift-console/console-ffd5bf477-69lqk" Sep 18 06:28:37.290493 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:37.290475 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25-console-config\") pod \"console-ffd5bf477-69lqk\" (UID: \"7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25\") " pod="openshift-console/console-ffd5bf477-69lqk" Sep 18 06:28:37.290771 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:37.290749 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25-trusted-ca-bundle\") pod \"console-ffd5bf477-69lqk\" (UID: \"7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25\") " pod="openshift-console/console-ffd5bf477-69lqk" Sep 18 06:28:37.292056 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:37.292039 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25-console-oauth-config\") pod \"console-ffd5bf477-69lqk\" (UID: \"7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25\") " pod="openshift-console/console-ffd5bf477-69lqk" Sep 18 06:28:37.292197 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:37.292179 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25-console-serving-cert\") pod \"console-ffd5bf477-69lqk\" (UID: \"7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25\") " pod="openshift-console/console-ffd5bf477-69lqk" Sep 18 06:28:37.296576 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:37.296554 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgvw2\" (UniqueName: \"kubernetes.io/projected/7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25-kube-api-access-dgvw2\") pod \"console-ffd5bf477-69lqk\" (UID: \"7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25\") " pod="openshift-console/console-ffd5bf477-69lqk" Sep 18 06:28:37.387844 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:37.387804 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ffd5bf477-69lqk" Sep 18 06:28:37.658794 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:37.658765 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-ffd5bf477-69lqk"] Sep 18 06:28:37.661403 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:28:37.661375 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c8b30a6_4fc7_4f7d_89c2_f5a2741eeb25.slice/crio-c4020b42a64e5f609b7d636bd99a7742c7226451af1e72402458d98fae05636c WatchSource:0}: Error finding container c4020b42a64e5f609b7d636bd99a7742c7226451af1e72402458d98fae05636c: Status 404 returned error can't find the container with id c4020b42a64e5f609b7d636bd99a7742c7226451af1e72402458d98fae05636c Sep 18 06:28:37.705860 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:37.705828 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ffd5bf477-69lqk" event={"ID":"7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25","Type":"ContainerStarted","Data":"c4020b42a64e5f609b7d636bd99a7742c7226451af1e72402458d98fae05636c"} Sep 18 06:28:38.711264 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:38.711229 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ffd5bf477-69lqk" event={"ID":"7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25","Type":"ContainerStarted","Data":"e565a49c46d43dc09adcd4ad9e0bca31934be082ac407caf0c5d6454d6d449fc"} Sep 18 06:28:38.727336 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:38.727282 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-ffd5bf477-69lqk" podStartSLOduration=1.7272651890000001 podStartE2EDuration="1.727265189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:28:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:28:38.726562216 +0000 UTC m=+380.465978421" watchObservedRunningTime="2026-09-18 06:28:38.727265189 +0000 UTC m=+380.466681404" Sep 18 06:28:47.388870 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:47.388831 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-ffd5bf477-69lqk" Sep 18 06:28:47.389291 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:47.388924 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-ffd5bf477-69lqk" Sep 18 06:28:47.393264 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:47.393239 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-ffd5bf477-69lqk" Sep 18 06:28:47.752080 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:28:47.751999 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-ffd5bf477-69lqk" Sep 18 06:30:11.360281 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:30:11.360248 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-ffd5bf477-69lqk"] Sep 18 06:30:36.381945 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:30:36.381907 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-ffd5bf477-69lqk" podUID="7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25" containerName="console" containerID="cri-o://e565a49c46d43dc09adcd4ad9e0bca31934be082ac407caf0c5d6454d6d449fc" gracePeriod=15 Sep 18 06:30:36.621330 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:30:36.621303 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ffd5bf477-69lqk" Sep 18 06:30:36.654072 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:30:36.654008 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgvw2\" (UniqueName: \"kubernetes.io/projected/7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25-kube-api-access-dgvw2\") pod \"7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25\" (UID: \"7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25\") " Sep 18 06:30:36.654072 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:30:36.654042 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25-oauth-serving-cert\") pod \"7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25\" (UID: \"7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25\") " Sep 18 06:30:36.654267 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:30:36.654074 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25-console-config\") pod \"7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25\" (UID: \"7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25\") " Sep 18 06:30:36.654267 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:30:36.654103 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25-service-ca\") pod \"7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25\" (UID: \"7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25\") " Sep 18 06:30:36.654267 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:30:36.654179 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25-console-oauth-config\") pod \"7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25\" (UID: \"7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25\") " Sep 18 06:30:36.654267 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:30:36.654231 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25-console-serving-cert\") pod \"7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25\" (UID: \"7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25\") " Sep 18 06:30:36.654501 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:30:36.654270 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25-trusted-ca-bundle\") pod \"7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25\" (UID: \"7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25\") " Sep 18 06:30:36.654501 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:30:36.654403 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25" (UID: "7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 06:30:36.654501 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:30:36.654466 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25-service-ca" (OuterVolumeSpecName: "service-ca") pod "7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25" (UID: "7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 06:30:36.654501 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:30:36.654478 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25-console-config" (OuterVolumeSpecName: "console-config") pod "7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25" (UID: "7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 06:30:36.654704 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:30:36.654536 2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25-oauth-serving-cert\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:30:36.654704 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:30:36.654553 2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25-console-config\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:30:36.654704 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:30:36.654566 2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25-service-ca\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:30:36.654814 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:30:36.654794 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25" (UID: "7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 06:30:36.656433 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:30:36.656382 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25" (UID: "7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 06:30:36.656543 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:30:36.656436 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25-kube-api-access-dgvw2" (OuterVolumeSpecName: "kube-api-access-dgvw2") pod "7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25" (UID: "7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25"). InnerVolumeSpecName "kube-api-access-dgvw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 06:30:36.656543 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:30:36.656485 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25" (UID: "7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 06:30:36.755160 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:30:36.755131 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgvw2\" (UniqueName: \"kubernetes.io/projected/7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25-kube-api-access-dgvw2\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:30:36.755160 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:30:36.755154 2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25-console-oauth-config\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:30:36.755160 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:30:36.755163 2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25-console-serving-cert\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:30:36.755350 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:30:36.755172 2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25-trusted-ca-bundle\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:30:37.164958 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:30:37.164923 2567 generic.go:356] "Generic (PLEG): container finished" podID="7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25" containerID="e565a49c46d43dc09adcd4ad9e0bca31934be082ac407caf0c5d6454d6d449fc" exitCode=0 Sep 18 06:30:37.165124 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:30:37.164965 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ffd5bf477-69lqk" Sep 18 06:30:37.165124 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:30:37.164980 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ffd5bf477-69lqk" event={"ID":"7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25","Type":"ContainerDied","Data":"e565a49c46d43dc09adcd4ad9e0bca31934be082ac407caf0c5d6454d6d449fc"} Sep 18 06:30:37.165124 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:30:37.165020 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ffd5bf477-69lqk" event={"ID":"7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25","Type":"ContainerDied","Data":"c4020b42a64e5f609b7d636bd99a7742c7226451af1e72402458d98fae05636c"} Sep 18 06:30:37.165124 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:30:37.165038 2567 scope.go:117] "RemoveContainer" containerID="e565a49c46d43dc09adcd4ad9e0bca31934be082ac407caf0c5d6454d6d449fc" Sep 18 06:30:37.174224 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:30:37.174206 2567 scope.go:117] "RemoveContainer" containerID="e565a49c46d43dc09adcd4ad9e0bca31934be082ac407caf0c5d6454d6d449fc" Sep 18 06:30:37.174494 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:30:37.174475 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e565a49c46d43dc09adcd4ad9e0bca31934be082ac407caf0c5d6454d6d449fc\": container with ID starting with e565a49c46d43dc09adcd4ad9e0bca31934be082ac407caf0c5d6454d6d449fc not found: ID does not exist" containerID="e565a49c46d43dc09adcd4ad9e0bca31934be082ac407caf0c5d6454d6d449fc" Sep 18 06:30:37.174564 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:30:37.174508 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e565a49c46d43dc09adcd4ad9e0bca31934be082ac407caf0c5d6454d6d449fc"} err="failed to get container status \"e565a49c46d43dc09adcd4ad9e0bca31934be082ac407caf0c5d6454d6d449fc\": rpc error: code = NotFound desc = could not find container \"e565a49c46d43dc09adcd4ad9e0bca31934be082ac407caf0c5d6454d6d449fc\": container with ID starting with e565a49c46d43dc09adcd4ad9e0bca31934be082ac407caf0c5d6454d6d449fc not found: ID does not exist" Sep 18 06:30:37.185174 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:30:37.185145 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-ffd5bf477-69lqk"] Sep 18 06:30:37.190740 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:30:37.190722 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-ffd5bf477-69lqk"] Sep 18 06:30:38.444551 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:30:38.444517 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25" path="/var/lib/kubelet/pods/7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25/volumes" Sep 18 06:30:53.290679 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:30:53.290641 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nhvh4"] Sep 18 06:30:53.291266 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:30:53.290952 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25" containerName="console" Sep 18 06:30:53.291266 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:30:53.290963 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25" containerName="console" Sep 18 06:30:53.291266 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:30:53.291017 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c8b30a6-4fc7-4f7d-89c2-f5a2741eeb25" containerName="console" Sep 18 06:30:53.295289 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:30:53.295272 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nhvh4" Sep 18 06:30:53.297880 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:30:53.297855 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 18 06:30:53.298001 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:30:53.297951 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-hg9g9\"" Sep 18 06:30:53.298753 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:30:53.298738 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 18 06:30:53.302266 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:30:53.302216 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nhvh4"] Sep 18 06:30:53.376776 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:30:53.376744 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twfq9\" (UniqueName: \"kubernetes.io/projected/2c545831-2d9d-480b-b14d-3a68e532963e-kube-api-access-twfq9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nhvh4\" (UID: \"2c545831-2d9d-480b-b14d-3a68e532963e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nhvh4" Sep 18 06:30:53.376947 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:30:53.376796 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c545831-2d9d-480b-b14d-3a68e532963e-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nhvh4\" (UID: \"2c545831-2d9d-480b-b14d-3a68e532963e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nhvh4" Sep 18 06:30:53.376947 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:30:53.376844 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c545831-2d9d-480b-b14d-3a68e532963e-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nhvh4\" (UID: \"2c545831-2d9d-480b-b14d-3a68e532963e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nhvh4" Sep 18 06:30:53.477743 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:30:53.477706 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c545831-2d9d-480b-b14d-3a68e532963e-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nhvh4\" (UID: \"2c545831-2d9d-480b-b14d-3a68e532963e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nhvh4" Sep 18 06:30:53.477743 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:30:53.477745 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c545831-2d9d-480b-b14d-3a68e532963e-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nhvh4\" (UID: \"2c545831-2d9d-480b-b14d-3a68e532963e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nhvh4" Sep 18 06:30:53.477966 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:30:53.477789 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twfq9\" (UniqueName: \"kubernetes.io/projected/2c545831-2d9d-480b-b14d-3a68e532963e-kube-api-access-twfq9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nhvh4\" (UID: \"2c545831-2d9d-480b-b14d-3a68e532963e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nhvh4" Sep 18 06:30:53.478083 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:30:53.478063 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c545831-2d9d-480b-b14d-3a68e532963e-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nhvh4\" (UID: \"2c545831-2d9d-480b-b14d-3a68e532963e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nhvh4" Sep 18 06:30:53.478168 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:30:53.478147 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c545831-2d9d-480b-b14d-3a68e532963e-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nhvh4\" (UID: \"2c545831-2d9d-480b-b14d-3a68e532963e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nhvh4" Sep 18 06:30:53.485968 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:30:53.485947 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twfq9\" (UniqueName: \"kubernetes.io/projected/2c545831-2d9d-480b-b14d-3a68e532963e-kube-api-access-twfq9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nhvh4\" (UID: \"2c545831-2d9d-480b-b14d-3a68e532963e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nhvh4" Sep 18 06:30:53.605092 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:30:53.605062 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nhvh4" Sep 18 06:30:53.872625 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:30:53.872548 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nhvh4"] Sep 18 06:30:53.875683 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:30:53.875661 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c545831_2d9d_480b_b14d_3a68e532963e.slice/crio-0d4c20e7a43b20237051d9201fcd34c805a07a11eb0feb2e1d5625f4d20050f9 WatchSource:0}: Error finding container 0d4c20e7a43b20237051d9201fcd34c805a07a11eb0feb2e1d5625f4d20050f9: Status 404 returned error can't find the container with id 0d4c20e7a43b20237051d9201fcd34c805a07a11eb0feb2e1d5625f4d20050f9 Sep 18 06:30:54.236285 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:30:54.236207 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nhvh4" event={"ID":"2c545831-2d9d-480b-b14d-3a68e532963e","Type":"ContainerStarted","Data":"0d4c20e7a43b20237051d9201fcd34c805a07a11eb0feb2e1d5625f4d20050f9"} Sep 18 06:31:01.270239 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:01.270208 2567 generic.go:356] "Generic (PLEG): container finished" podID="2c545831-2d9d-480b-b14d-3a68e532963e" containerID="01bc220f1ea57bc39accfe31d97af4960f1c3648b95919f54983e5ec89bb1c8b" exitCode=0 Sep 18 06:31:01.270630 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:01.270278 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nhvh4" event={"ID":"2c545831-2d9d-480b-b14d-3a68e532963e","Type":"ContainerDied","Data":"01bc220f1ea57bc39accfe31d97af4960f1c3648b95919f54983e5ec89bb1c8b"} Sep 18 06:31:04.284829 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:04.284788 2567 generic.go:356] "Generic (PLEG): container finished" podID="2c545831-2d9d-480b-b14d-3a68e532963e" containerID="196cf527336735a14d177f2d24922aae74e309f4d0dda82b5a951194f9973516" exitCode=0 Sep 18 06:31:04.285221 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:04.284847 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nhvh4" event={"ID":"2c545831-2d9d-480b-b14d-3a68e532963e","Type":"ContainerDied","Data":"196cf527336735a14d177f2d24922aae74e309f4d0dda82b5a951194f9973516"} Sep 18 06:31:12.320292 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:12.320248 2567 generic.go:356] "Generic (PLEG): container finished" podID="2c545831-2d9d-480b-b14d-3a68e532963e" containerID="68ac1e1ed98a9f5dfe281dbe610e72ffebf5f5b89c761dc762f1d9d37bc143c1" exitCode=0 Sep 18 06:31:12.320684 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:12.320334 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nhvh4" event={"ID":"2c545831-2d9d-480b-b14d-3a68e532963e","Type":"ContainerDied","Data":"68ac1e1ed98a9f5dfe281dbe610e72ffebf5f5b89c761dc762f1d9d37bc143c1"} Sep 18 06:31:13.452645 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:13.452610 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nhvh4" Sep 18 06:31:13.548361 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:13.548328 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c545831-2d9d-480b-b14d-3a68e532963e-util\") pod \"2c545831-2d9d-480b-b14d-3a68e532963e\" (UID: \"2c545831-2d9d-480b-b14d-3a68e532963e\") " Sep 18 06:31:13.548567 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:13.548411 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c545831-2d9d-480b-b14d-3a68e532963e-bundle\") pod \"2c545831-2d9d-480b-b14d-3a68e532963e\" (UID: \"2c545831-2d9d-480b-b14d-3a68e532963e\") " Sep 18 06:31:13.548567 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:13.548439 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-twfq9\" (UniqueName: \"kubernetes.io/projected/2c545831-2d9d-480b-b14d-3a68e532963e-kube-api-access-twfq9\") pod \"2c545831-2d9d-480b-b14d-3a68e532963e\" (UID: \"2c545831-2d9d-480b-b14d-3a68e532963e\") " Sep 18 06:31:13.549076 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:13.549049 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c545831-2d9d-480b-b14d-3a68e532963e-bundle" (OuterVolumeSpecName: "bundle") pod "2c545831-2d9d-480b-b14d-3a68e532963e" (UID: "2c545831-2d9d-480b-b14d-3a68e532963e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:31:13.550647 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:13.550624 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c545831-2d9d-480b-b14d-3a68e532963e-kube-api-access-twfq9" (OuterVolumeSpecName: "kube-api-access-twfq9") pod "2c545831-2d9d-480b-b14d-3a68e532963e" (UID: "2c545831-2d9d-480b-b14d-3a68e532963e"). InnerVolumeSpecName "kube-api-access-twfq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 06:31:13.552473 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:13.552453 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c545831-2d9d-480b-b14d-3a68e532963e-util" (OuterVolumeSpecName: "util") pod "2c545831-2d9d-480b-b14d-3a68e532963e" (UID: "2c545831-2d9d-480b-b14d-3a68e532963e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:31:13.649833 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:13.649795 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c545831-2d9d-480b-b14d-3a68e532963e-util\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:31:13.649833 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:13.649824 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c545831-2d9d-480b-b14d-3a68e532963e-bundle\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:31:13.649833 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:13.649833 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twfq9\" (UniqueName: \"kubernetes.io/projected/2c545831-2d9d-480b-b14d-3a68e532963e-kube-api-access-twfq9\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:31:14.329977 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:14.329947 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nhvh4" Sep 18 06:31:14.330184 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:14.329954 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nhvh4" event={"ID":"2c545831-2d9d-480b-b14d-3a68e532963e","Type":"ContainerDied","Data":"0d4c20e7a43b20237051d9201fcd34c805a07a11eb0feb2e1d5625f4d20050f9"} Sep 18 06:31:14.330184 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:14.330096 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d4c20e7a43b20237051d9201fcd34c805a07a11eb0feb2e1d5625f4d20050f9" Sep 18 06:31:21.212474 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:21.212436 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bdnpc"] Sep 18 06:31:21.212936 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:21.212876 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c545831-2d9d-480b-b14d-3a68e532963e" containerName="extract" Sep 18 06:31:21.212936 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:21.212893 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c545831-2d9d-480b-b14d-3a68e532963e" containerName="extract" Sep 18 06:31:21.212936 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:21.212914 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c545831-2d9d-480b-b14d-3a68e532963e" containerName="pull" Sep 18 06:31:21.212936 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:21.212923 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c545831-2d9d-480b-b14d-3a68e532963e" containerName="pull" Sep 18 06:31:21.213140 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:21.212951 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c545831-2d9d-480b-b14d-3a68e532963e" containerName="util" Sep 18 06:31:21.213140 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:21.212959 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c545831-2d9d-480b-b14d-3a68e532963e" containerName="util" Sep 18 06:31:21.213140 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:21.213040 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c545831-2d9d-480b-b14d-3a68e532963e" containerName="extract" Sep 18 06:31:21.217273 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:21.217252 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bdnpc" Sep 18 06:31:21.219912 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:21.219890 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-gvcc7\"" Sep 18 06:31:21.220012 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:21.219937 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Sep 18 06:31:21.220012 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:21.219967 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Sep 18 06:31:21.235261 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:21.235241 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bdnpc"] Sep 18 06:31:21.313383 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:21.313340 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08c77ca1-6013-4d16-a45a-f48a1c21a314-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-bdnpc\" (UID: \"08c77ca1-6013-4d16-a45a-f48a1c21a314\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bdnpc" Sep 18 06:31:21.313506 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:21.313411 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wfp6\" (UniqueName: \"kubernetes.io/projected/08c77ca1-6013-4d16-a45a-f48a1c21a314-kube-api-access-7wfp6\") pod \"cert-manager-operator-controller-manager-55c444bf86-bdnpc\" (UID: \"08c77ca1-6013-4d16-a45a-f48a1c21a314\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bdnpc" Sep 18 06:31:21.414758 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:21.414729 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08c77ca1-6013-4d16-a45a-f48a1c21a314-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-bdnpc\" (UID: \"08c77ca1-6013-4d16-a45a-f48a1c21a314\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bdnpc" Sep 18 06:31:21.414899 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:21.414781 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wfp6\" (UniqueName: \"kubernetes.io/projected/08c77ca1-6013-4d16-a45a-f48a1c21a314-kube-api-access-7wfp6\") pod \"cert-manager-operator-controller-manager-55c444bf86-bdnpc\" (UID: \"08c77ca1-6013-4d16-a45a-f48a1c21a314\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bdnpc" Sep 18 06:31:21.415128 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:21.415104 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08c77ca1-6013-4d16-a45a-f48a1c21a314-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-bdnpc\" (UID: \"08c77ca1-6013-4d16-a45a-f48a1c21a314\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bdnpc" Sep 18 06:31:21.422519 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:21.422498 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wfp6\" (UniqueName: \"kubernetes.io/projected/08c77ca1-6013-4d16-a45a-f48a1c21a314-kube-api-access-7wfp6\") pod \"cert-manager-operator-controller-manager-55c444bf86-bdnpc\" (UID: \"08c77ca1-6013-4d16-a45a-f48a1c21a314\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bdnpc" Sep 18 06:31:21.526766 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:21.526710 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bdnpc" Sep 18 06:31:21.799360 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:21.799311 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bdnpc"] Sep 18 06:31:21.802442 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:31:21.802413 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08c77ca1_6013_4d16_a45a_f48a1c21a314.slice/crio-8c02f0c28542aea626ed3edd736cb0f980ffc4f6de78cc9671e182f39a9ae7f7 WatchSource:0}: Error finding container 8c02f0c28542aea626ed3edd736cb0f980ffc4f6de78cc9671e182f39a9ae7f7: Status 404 returned error can't find the container with id 8c02f0c28542aea626ed3edd736cb0f980ffc4f6de78cc9671e182f39a9ae7f7 Sep 18 06:31:22.362995 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:22.362964 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bdnpc" event={"ID":"08c77ca1-6013-4d16-a45a-f48a1c21a314","Type":"ContainerStarted","Data":"8c02f0c28542aea626ed3edd736cb0f980ffc4f6de78cc9671e182f39a9ae7f7"} Sep 18 06:31:24.379816 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:24.379780 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bdnpc" event={"ID":"08c77ca1-6013-4d16-a45a-f48a1c21a314","Type":"ContainerStarted","Data":"f68184209e9f0e3ecade6dd008d37bc50d56a1e3f54fbf00e4290238e9f6e508"} Sep 18 06:31:24.400064 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:24.399861 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bdnpc" podStartSLOduration=1.171771308 podStartE2EDuration="3.399842985s" totalImagesPullingTime="2.228071677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:31:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:31:24.397010536 +0000 UTC m=+546.136426722" watchObservedRunningTime="2026-09-18 06:31:24.399842985 +0000 UTC m=+546.139259159" Sep 18 06:31:25.539329 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:25.539298 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f86x2p"] Sep 18 06:31:25.541719 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:25.541697 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f86x2p" Sep 18 06:31:25.544474 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:25.544453 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 18 06:31:25.544586 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:25.544473 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 18 06:31:25.545493 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:25.545475 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-hg9g9\"" Sep 18 06:31:25.551739 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:25.551712 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f86x2p"] Sep 18 06:31:25.645916 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:25.645887 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c7c1b0dd-c278-4bc4-bc0c-b916a1e65592-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f86x2p\" (UID: \"c7c1b0dd-c278-4bc4-bc0c-b916a1e65592\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f86x2p" Sep 18 06:31:25.646058 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:25.645921 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24bt8\" (UniqueName: \"kubernetes.io/projected/c7c1b0dd-c278-4bc4-bc0c-b916a1e65592-kube-api-access-24bt8\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f86x2p\" (UID: \"c7c1b0dd-c278-4bc4-bc0c-b916a1e65592\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f86x2p" Sep 18 06:31:25.646058 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:25.645958 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c7c1b0dd-c278-4bc4-bc0c-b916a1e65592-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f86x2p\" (UID: \"c7c1b0dd-c278-4bc4-bc0c-b916a1e65592\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f86x2p" Sep 18 06:31:25.746422 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:25.746397 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c7c1b0dd-c278-4bc4-bc0c-b916a1e65592-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f86x2p\" (UID: \"c7c1b0dd-c278-4bc4-bc0c-b916a1e65592\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f86x2p" Sep 18 06:31:25.746559 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:25.746430 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24bt8\" (UniqueName: \"kubernetes.io/projected/c7c1b0dd-c278-4bc4-bc0c-b916a1e65592-kube-api-access-24bt8\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f86x2p\" (UID: \"c7c1b0dd-c278-4bc4-bc0c-b916a1e65592\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f86x2p" Sep 18 06:31:25.746559 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:25.746458 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c7c1b0dd-c278-4bc4-bc0c-b916a1e65592-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f86x2p\" (UID: \"c7c1b0dd-c278-4bc4-bc0c-b916a1e65592\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f86x2p" Sep 18 06:31:25.746783 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:25.746755 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c7c1b0dd-c278-4bc4-bc0c-b916a1e65592-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f86x2p\" (UID: \"c7c1b0dd-c278-4bc4-bc0c-b916a1e65592\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f86x2p" Sep 18 06:31:25.746825 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:25.746777 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c7c1b0dd-c278-4bc4-bc0c-b916a1e65592-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f86x2p\" (UID: \"c7c1b0dd-c278-4bc4-bc0c-b916a1e65592\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f86x2p" Sep 18 06:31:25.754292 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:25.754269 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24bt8\" (UniqueName: \"kubernetes.io/projected/c7c1b0dd-c278-4bc4-bc0c-b916a1e65592-kube-api-access-24bt8\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f86x2p\" (UID: \"c7c1b0dd-c278-4bc4-bc0c-b916a1e65592\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f86x2p" Sep 18 06:31:25.851558 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:25.851529 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f86x2p" Sep 18 06:31:26.120405 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:26.120360 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f86x2p"] Sep 18 06:31:26.122887 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:31:26.122861 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7c1b0dd_c278_4bc4_bc0c_b916a1e65592.slice/crio-b465410a11d99cd539168d62674c05a8b95c2164f5cf54e7b1eeb70593898d1a WatchSource:0}: Error finding container b465410a11d99cd539168d62674c05a8b95c2164f5cf54e7b1eeb70593898d1a: Status 404 returned error can't find the container with id b465410a11d99cd539168d62674c05a8b95c2164f5cf54e7b1eeb70593898d1a Sep 18 06:31:26.390225 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:26.390160 2567 generic.go:356] "Generic (PLEG): container finished" podID="c7c1b0dd-c278-4bc4-bc0c-b916a1e65592" containerID="179a44409ccd6fd52fed81bde814f46f5331ba990c393b924ce5cb0c6214055e" exitCode=0 Sep 18 06:31:26.390475 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:26.390245 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f86x2p" event={"ID":"c7c1b0dd-c278-4bc4-bc0c-b916a1e65592","Type":"ContainerDied","Data":"179a44409ccd6fd52fed81bde814f46f5331ba990c393b924ce5cb0c6214055e"} Sep 18 06:31:26.390475 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:26.390290 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f86x2p" event={"ID":"c7c1b0dd-c278-4bc4-bc0c-b916a1e65592","Type":"ContainerStarted","Data":"b465410a11d99cd539168d62674c05a8b95c2164f5cf54e7b1eeb70593898d1a"} Sep 18 06:31:27.332691 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:27.332659 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-rbzz6"] Sep 18 06:31:27.334919 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:27.334898 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-rbzz6" Sep 18 06:31:27.338096 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:27.338077 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 18 06:31:27.339094 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:27.339080 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 18 06:31:27.339164 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:27.339083 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-7qnf2\"" Sep 18 06:31:27.392333 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:27.392298 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-rbzz6"] Sep 18 06:31:27.461205 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:27.461177 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8e3fccf9-189d-4e15-ac1e-5e87d5a49aa3-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-rbzz6\" (UID: \"8e3fccf9-189d-4e15-ac1e-5e87d5a49aa3\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-rbzz6" Sep 18 06:31:27.461346 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:27.461248 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqh9t\" (UniqueName: \"kubernetes.io/projected/8e3fccf9-189d-4e15-ac1e-5e87d5a49aa3-kube-api-access-lqh9t\") pod \"cert-manager-cainjector-559cdd8c87-rbzz6\" (UID: \"8e3fccf9-189d-4e15-ac1e-5e87d5a49aa3\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-rbzz6" Sep 18 06:31:27.562276 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:27.562247 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqh9t\" (UniqueName: \"kubernetes.io/projected/8e3fccf9-189d-4e15-ac1e-5e87d5a49aa3-kube-api-access-lqh9t\") pod \"cert-manager-cainjector-559cdd8c87-rbzz6\" (UID: \"8e3fccf9-189d-4e15-ac1e-5e87d5a49aa3\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-rbzz6" Sep 18 06:31:27.562446 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:27.562320 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8e3fccf9-189d-4e15-ac1e-5e87d5a49aa3-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-rbzz6\" (UID: \"8e3fccf9-189d-4e15-ac1e-5e87d5a49aa3\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-rbzz6" Sep 18 06:31:27.570833 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:27.570805 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8e3fccf9-189d-4e15-ac1e-5e87d5a49aa3-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-rbzz6\" (UID: \"8e3fccf9-189d-4e15-ac1e-5e87d5a49aa3\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-rbzz6" Sep 18 06:31:27.571030 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:27.571008 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqh9t\" (UniqueName: \"kubernetes.io/projected/8e3fccf9-189d-4e15-ac1e-5e87d5a49aa3-kube-api-access-lqh9t\") pod \"cert-manager-cainjector-559cdd8c87-rbzz6\" (UID: \"8e3fccf9-189d-4e15-ac1e-5e87d5a49aa3\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-rbzz6" Sep 18 06:31:27.643127 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:27.643068 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-rbzz6" Sep 18 06:31:27.916897 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:27.916872 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-rbzz6"] Sep 18 06:31:27.919516 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:31:27.919484 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e3fccf9_189d_4e15_ac1e_5e87d5a49aa3.slice/crio-ca6b0268bd0e5648fa9e7d573af29909a94860de6585abefdb59b40e632e62ee WatchSource:0}: Error finding container ca6b0268bd0e5648fa9e7d573af29909a94860de6585abefdb59b40e632e62ee: Status 404 returned error can't find the container with id ca6b0268bd0e5648fa9e7d573af29909a94860de6585abefdb59b40e632e62ee Sep 18 06:31:28.403067 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:28.403033 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-rbzz6" event={"ID":"8e3fccf9-189d-4e15-ac1e-5e87d5a49aa3","Type":"ContainerStarted","Data":"ca6b0268bd0e5648fa9e7d573af29909a94860de6585abefdb59b40e632e62ee"} Sep 18 06:31:29.415243 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:29.415207 2567 generic.go:356] "Generic (PLEG): container finished" podID="c7c1b0dd-c278-4bc4-bc0c-b916a1e65592" containerID="5fad3c7a6bafea330f683175554f823a2605e9954b24d44addefe0a3efeeb540" exitCode=0 Sep 18 06:31:29.415725 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:29.415288 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f86x2p" event={"ID":"c7c1b0dd-c278-4bc4-bc0c-b916a1e65592","Type":"ContainerDied","Data":"5fad3c7a6bafea330f683175554f823a2605e9954b24d44addefe0a3efeeb540"} Sep 18 06:31:30.424104 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:30.424067 2567 generic.go:356] "Generic (PLEG): container finished" podID="c7c1b0dd-c278-4bc4-bc0c-b916a1e65592" containerID="316b79e8aab93efee1b9b5253a9bfb2b21c1596360f71f5441d7ff4274187f97" exitCode=0 Sep 18 06:31:30.424589 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:30.424148 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f86x2p" event={"ID":"c7c1b0dd-c278-4bc4-bc0c-b916a1e65592","Type":"ContainerDied","Data":"316b79e8aab93efee1b9b5253a9bfb2b21c1596360f71f5441d7ff4274187f97"} Sep 18 06:31:31.430692 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:31.430654 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-rbzz6" event={"ID":"8e3fccf9-189d-4e15-ac1e-5e87d5a49aa3","Type":"ContainerStarted","Data":"faaf6fd63a30b3ad63131fdc87a65f88e0d373dbe841124a671815fb661c7ee1"} Sep 18 06:31:31.445284 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:31.445238 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-rbzz6" podStartSLOduration=1.784882612 podStartE2EDuration="4.445223271s" totalImagesPullingTime="2.660340659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:31:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:31:31.444577202 +0000 UTC m=+553.183993394" watchObservedRunningTime="2026-09-18 06:31:31.445223271 +0000 UTC m=+553.184639445" Sep 18 06:31:31.552297 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:31.552272 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f86x2p" Sep 18 06:31:31.693426 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:31.693343 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-24bt8\" (UniqueName: \"kubernetes.io/projected/c7c1b0dd-c278-4bc4-bc0c-b916a1e65592-kube-api-access-24bt8\") pod \"c7c1b0dd-c278-4bc4-bc0c-b916a1e65592\" (UID: \"c7c1b0dd-c278-4bc4-bc0c-b916a1e65592\") " Sep 18 06:31:31.693426 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:31.693421 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c7c1b0dd-c278-4bc4-bc0c-b916a1e65592-bundle\") pod \"c7c1b0dd-c278-4bc4-bc0c-b916a1e65592\" (UID: \"c7c1b0dd-c278-4bc4-bc0c-b916a1e65592\") " Sep 18 06:31:31.693639 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:31.693478 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c7c1b0dd-c278-4bc4-bc0c-b916a1e65592-util\") pod \"c7c1b0dd-c278-4bc4-bc0c-b916a1e65592\" (UID: \"c7c1b0dd-c278-4bc4-bc0c-b916a1e65592\") " Sep 18 06:31:31.693774 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:31.693751 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7c1b0dd-c278-4bc4-bc0c-b916a1e65592-bundle" (OuterVolumeSpecName: "bundle") pod "c7c1b0dd-c278-4bc4-bc0c-b916a1e65592" (UID: "c7c1b0dd-c278-4bc4-bc0c-b916a1e65592"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:31:31.695309 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:31.695289 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7c1b0dd-c278-4bc4-bc0c-b916a1e65592-kube-api-access-24bt8" (OuterVolumeSpecName: "kube-api-access-24bt8") pod "c7c1b0dd-c278-4bc4-bc0c-b916a1e65592" (UID: "c7c1b0dd-c278-4bc4-bc0c-b916a1e65592"). InnerVolumeSpecName "kube-api-access-24bt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 06:31:31.697632 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:31.697595 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7c1b0dd-c278-4bc4-bc0c-b916a1e65592-util" (OuterVolumeSpecName: "util") pod "c7c1b0dd-c278-4bc4-bc0c-b916a1e65592" (UID: "c7c1b0dd-c278-4bc4-bc0c-b916a1e65592"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:31:31.794967 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:31.794934 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24bt8\" (UniqueName: \"kubernetes.io/projected/c7c1b0dd-c278-4bc4-bc0c-b916a1e65592-kube-api-access-24bt8\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:31:31.794967 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:31.794965 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c7c1b0dd-c278-4bc4-bc0c-b916a1e65592-bundle\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:31:31.794967 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:31.794974 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c7c1b0dd-c278-4bc4-bc0c-b916a1e65592-util\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:31:32.069656 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:32.069572 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-ftbn4"] Sep 18 06:31:32.069968 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:32.069950 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7c1b0dd-c278-4bc4-bc0c-b916a1e65592" containerName="extract" Sep 18 06:31:32.070050 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:32.069972 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7c1b0dd-c278-4bc4-bc0c-b916a1e65592" containerName="extract" Sep 18 06:31:32.070050 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:32.069991 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7c1b0dd-c278-4bc4-bc0c-b916a1e65592" containerName="util" Sep 18 06:31:32.070050 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:32.069999 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7c1b0dd-c278-4bc4-bc0c-b916a1e65592" containerName="util" Sep 18 06:31:32.070050 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:32.070012 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7c1b0dd-c278-4bc4-bc0c-b916a1e65592" containerName="pull" Sep 18 06:31:32.070050 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:32.070021 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7c1b0dd-c278-4bc4-bc0c-b916a1e65592" containerName="pull" Sep 18 06:31:32.070283 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:32.070100 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7c1b0dd-c278-4bc4-bc0c-b916a1e65592" containerName="extract" Sep 18 06:31:32.075707 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:32.075679 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-ftbn4" Sep 18 06:31:32.078889 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:32.078864 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-6hctw\"" Sep 18 06:31:32.079914 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:32.079893 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-ftbn4"] Sep 18 06:31:32.198122 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:32.198087 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/08366a9b-14f0-4808-81a4-19c723790e28-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-ftbn4\" (UID: \"08366a9b-14f0-4808-81a4-19c723790e28\") " pod="cert-manager/cert-manager-webhook-6486ffd997-ftbn4" Sep 18 06:31:32.198285 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:32.198141 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdzcs\" (UniqueName: \"kubernetes.io/projected/08366a9b-14f0-4808-81a4-19c723790e28-kube-api-access-vdzcs\") pod \"cert-manager-webhook-6486ffd997-ftbn4\" (UID: \"08366a9b-14f0-4808-81a4-19c723790e28\") " pod="cert-manager/cert-manager-webhook-6486ffd997-ftbn4" Sep 18 06:31:32.299022 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:32.298995 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/08366a9b-14f0-4808-81a4-19c723790e28-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-ftbn4\" (UID: \"08366a9b-14f0-4808-81a4-19c723790e28\") " pod="cert-manager/cert-manager-webhook-6486ffd997-ftbn4" Sep 18 06:31:32.299116 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:32.299053 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdzcs\" (UniqueName: \"kubernetes.io/projected/08366a9b-14f0-4808-81a4-19c723790e28-kube-api-access-vdzcs\") pod \"cert-manager-webhook-6486ffd997-ftbn4\" (UID: \"08366a9b-14f0-4808-81a4-19c723790e28\") " pod="cert-manager/cert-manager-webhook-6486ffd997-ftbn4" Sep 18 06:31:32.308379 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:32.308343 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/08366a9b-14f0-4808-81a4-19c723790e28-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-ftbn4\" (UID: \"08366a9b-14f0-4808-81a4-19c723790e28\") " pod="cert-manager/cert-manager-webhook-6486ffd997-ftbn4" Sep 18 06:31:32.308494 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:32.308479 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdzcs\" (UniqueName: \"kubernetes.io/projected/08366a9b-14f0-4808-81a4-19c723790e28-kube-api-access-vdzcs\") pod \"cert-manager-webhook-6486ffd997-ftbn4\" (UID: \"08366a9b-14f0-4808-81a4-19c723790e28\") " pod="cert-manager/cert-manager-webhook-6486ffd997-ftbn4" Sep 18 06:31:32.385492 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:32.385469 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-ftbn4" Sep 18 06:31:32.441021 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:32.439511 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f86x2p" Sep 18 06:31:32.447826 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:32.447768 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f86x2p" event={"ID":"c7c1b0dd-c278-4bc4-bc0c-b916a1e65592","Type":"ContainerDied","Data":"b465410a11d99cd539168d62674c05a8b95c2164f5cf54e7b1eeb70593898d1a"} Sep 18 06:31:32.447826 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:32.447818 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b465410a11d99cd539168d62674c05a8b95c2164f5cf54e7b1eeb70593898d1a" Sep 18 06:31:32.654523 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:32.654453 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-ftbn4"] Sep 18 06:31:32.657492 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:31:32.657465 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod08366a9b_14f0_4808_81a4_19c723790e28.slice/crio-c79dfe406dc0f9ff037cdc1a7ea478e1834808ca551a0f560246d3391c876389 WatchSource:0}: Error finding container c79dfe406dc0f9ff037cdc1a7ea478e1834808ca551a0f560246d3391c876389: Status 404 returned error can't find the container with id c79dfe406dc0f9ff037cdc1a7ea478e1834808ca551a0f560246d3391c876389 Sep 18 06:31:33.445422 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:33.445391 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-ftbn4" event={"ID":"08366a9b-14f0-4808-81a4-19c723790e28","Type":"ContainerStarted","Data":"ff20965bfd0c4ce7e30595564261daed157804d1b9531500cabae4373e5df74d"} Sep 18 06:31:33.445422 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:33.445428 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-ftbn4" event={"ID":"08366a9b-14f0-4808-81a4-19c723790e28","Type":"ContainerStarted","Data":"c79dfe406dc0f9ff037cdc1a7ea478e1834808ca551a0f560246d3391c876389"} Sep 18 06:31:33.445833 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:33.445461 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-ftbn4" Sep 18 06:31:33.462805 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:33.462752 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-ftbn4" podStartSLOduration=1.462737539 podStartE2EDuration="1.462737539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:31:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:31:33.461128368 +0000 UTC m=+555.200544541" watchObservedRunningTime="2026-09-18 06:31:33.462737539 +0000 UTC m=+555.202153712" Sep 18 06:31:39.451486 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:39.451457 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-ftbn4" Sep 18 06:31:45.138435 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:45.138400 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7bgv7"] Sep 18 06:31:45.143296 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:45.143275 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7bgv7" Sep 18 06:31:45.145620 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:45.145598 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-hg9g9\"" Sep 18 06:31:45.145741 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:45.145635 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 18 06:31:45.146855 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:45.146834 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 18 06:31:45.148633 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:45.148616 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7bgv7"] Sep 18 06:31:45.191400 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:45.191382 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ddfb5d76-3a4b-45e0-8043-822b19afa29f-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7bgv7\" (UID: \"ddfb5d76-3a4b-45e0-8043-822b19afa29f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7bgv7" Sep 18 06:31:45.191496 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:45.191420 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ddfb5d76-3a4b-45e0-8043-822b19afa29f-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7bgv7\" (UID: \"ddfb5d76-3a4b-45e0-8043-822b19afa29f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7bgv7" Sep 18 06:31:45.191496 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:45.191466 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb48w\" (UniqueName: \"kubernetes.io/projected/ddfb5d76-3a4b-45e0-8043-822b19afa29f-kube-api-access-qb48w\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7bgv7\" (UID: \"ddfb5d76-3a4b-45e0-8043-822b19afa29f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7bgv7" Sep 18 06:31:45.292168 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:45.292143 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ddfb5d76-3a4b-45e0-8043-822b19afa29f-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7bgv7\" (UID: \"ddfb5d76-3a4b-45e0-8043-822b19afa29f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7bgv7" Sep 18 06:31:45.292397 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:45.292379 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ddfb5d76-3a4b-45e0-8043-822b19afa29f-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7bgv7\" (UID: \"ddfb5d76-3a4b-45e0-8043-822b19afa29f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7bgv7" Sep 18 06:31:45.292561 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:45.292545 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb48w\" (UniqueName: \"kubernetes.io/projected/ddfb5d76-3a4b-45e0-8043-822b19afa29f-kube-api-access-qb48w\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7bgv7\" (UID: \"ddfb5d76-3a4b-45e0-8043-822b19afa29f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7bgv7" Sep 18 06:31:45.292689 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:45.292550 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ddfb5d76-3a4b-45e0-8043-822b19afa29f-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7bgv7\" (UID: \"ddfb5d76-3a4b-45e0-8043-822b19afa29f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7bgv7" Sep 18 06:31:45.293036 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:45.293013 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ddfb5d76-3a4b-45e0-8043-822b19afa29f-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7bgv7\" (UID: \"ddfb5d76-3a4b-45e0-8043-822b19afa29f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7bgv7" Sep 18 06:31:45.303332 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:45.303310 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb48w\" (UniqueName: \"kubernetes.io/projected/ddfb5d76-3a4b-45e0-8043-822b19afa29f-kube-api-access-qb48w\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7bgv7\" (UID: \"ddfb5d76-3a4b-45e0-8043-822b19afa29f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7bgv7" Sep 18 06:31:45.453359 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:45.453275 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7bgv7" Sep 18 06:31:45.720719 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:45.720694 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7bgv7"] Sep 18 06:31:45.722482 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:31:45.722453 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddfb5d76_3a4b_45e0_8043_822b19afa29f.slice/crio-408523a80ce5ab1f5e69180f997955585522148fd2bec8ecfb48bb355a79818a WatchSource:0}: Error finding container 408523a80ce5ab1f5e69180f997955585522148fd2bec8ecfb48bb355a79818a: Status 404 returned error can't find the container with id 408523a80ce5ab1f5e69180f997955585522148fd2bec8ecfb48bb355a79818a Sep 18 06:31:46.502291 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:46.502255 2567 generic.go:356] "Generic (PLEG): container finished" podID="ddfb5d76-3a4b-45e0-8043-822b19afa29f" containerID="996f7a1d52abc3cd9138ffcda3ec253b1e7ea4ac92dd5782d4678123e63f79d1" exitCode=0 Sep 18 06:31:46.502669 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:46.502314 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7bgv7" event={"ID":"ddfb5d76-3a4b-45e0-8043-822b19afa29f","Type":"ContainerDied","Data":"996f7a1d52abc3cd9138ffcda3ec253b1e7ea4ac92dd5782d4678123e63f79d1"} Sep 18 06:31:46.502669 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:46.502381 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7bgv7" event={"ID":"ddfb5d76-3a4b-45e0-8043-822b19afa29f","Type":"ContainerStarted","Data":"408523a80ce5ab1f5e69180f997955585522148fd2bec8ecfb48bb355a79818a"} Sep 18 06:31:47.508708 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:47.508622 2567 generic.go:356] "Generic (PLEG): container finished" podID="ddfb5d76-3a4b-45e0-8043-822b19afa29f" containerID="b0997ec40fc83ac44c6b4e2f32263c84fdb5c919c3d0e0693df0f344f9b0d039" exitCode=0 Sep 18 06:31:47.509049 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:47.508709 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7bgv7" event={"ID":"ddfb5d76-3a4b-45e0-8043-822b19afa29f","Type":"ContainerDied","Data":"b0997ec40fc83ac44c6b4e2f32263c84fdb5c919c3d0e0693df0f344f9b0d039"} Sep 18 06:31:48.514909 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:48.514880 2567 generic.go:356] "Generic (PLEG): container finished" podID="ddfb5d76-3a4b-45e0-8043-822b19afa29f" containerID="bcf10b66c5a8370e4a614064bb1197a7bdf4454a5840dbe17cd45c0ced188c02" exitCode=0 Sep 18 06:31:48.515284 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:48.514946 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7bgv7" event={"ID":"ddfb5d76-3a4b-45e0-8043-822b19afa29f","Type":"ContainerDied","Data":"bcf10b66c5a8370e4a614064bb1197a7bdf4454a5840dbe17cd45c0ced188c02"} Sep 18 06:31:49.639743 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:49.639720 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7bgv7" Sep 18 06:31:49.721324 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:49.721296 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qb48w\" (UniqueName: \"kubernetes.io/projected/ddfb5d76-3a4b-45e0-8043-822b19afa29f-kube-api-access-qb48w\") pod \"ddfb5d76-3a4b-45e0-8043-822b19afa29f\" (UID: \"ddfb5d76-3a4b-45e0-8043-822b19afa29f\") " Sep 18 06:31:49.721479 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:49.721343 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ddfb5d76-3a4b-45e0-8043-822b19afa29f-bundle\") pod \"ddfb5d76-3a4b-45e0-8043-822b19afa29f\" (UID: \"ddfb5d76-3a4b-45e0-8043-822b19afa29f\") " Sep 18 06:31:49.721479 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:49.721387 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ddfb5d76-3a4b-45e0-8043-822b19afa29f-util\") pod \"ddfb5d76-3a4b-45e0-8043-822b19afa29f\" (UID: \"ddfb5d76-3a4b-45e0-8043-822b19afa29f\") " Sep 18 06:31:49.722072 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:49.722049 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddfb5d76-3a4b-45e0-8043-822b19afa29f-bundle" (OuterVolumeSpecName: "bundle") pod "ddfb5d76-3a4b-45e0-8043-822b19afa29f" (UID: "ddfb5d76-3a4b-45e0-8043-822b19afa29f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:31:49.723407 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:49.723354 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddfb5d76-3a4b-45e0-8043-822b19afa29f-kube-api-access-qb48w" (OuterVolumeSpecName: "kube-api-access-qb48w") pod "ddfb5d76-3a4b-45e0-8043-822b19afa29f" (UID: "ddfb5d76-3a4b-45e0-8043-822b19afa29f"). InnerVolumeSpecName "kube-api-access-qb48w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 06:31:49.726579 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:49.726558 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddfb5d76-3a4b-45e0-8043-822b19afa29f-util" (OuterVolumeSpecName: "util") pod "ddfb5d76-3a4b-45e0-8043-822b19afa29f" (UID: "ddfb5d76-3a4b-45e0-8043-822b19afa29f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:31:49.822057 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:49.821982 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb48w\" (UniqueName: \"kubernetes.io/projected/ddfb5d76-3a4b-45e0-8043-822b19afa29f-kube-api-access-qb48w\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:31:49.822057 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:49.822009 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ddfb5d76-3a4b-45e0-8043-822b19afa29f-bundle\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:31:49.822057 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:49.822023 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ddfb5d76-3a4b-45e0-8043-822b19afa29f-util\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:31:50.525388 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:50.525341 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7bgv7" event={"ID":"ddfb5d76-3a4b-45e0-8043-822b19afa29f","Type":"ContainerDied","Data":"408523a80ce5ab1f5e69180f997955585522148fd2bec8ecfb48bb355a79818a"} Sep 18 06:31:50.525553 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:50.525399 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7bgv7" Sep 18 06:31:50.525553 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:50.525433 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="408523a80ce5ab1f5e69180f997955585522148fd2bec8ecfb48bb355a79818a" Sep 18 06:31:53.572078 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:53.572047 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-64c86f57d4-cmwvd"] Sep 18 06:31:53.572477 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:53.572338 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddfb5d76-3a4b-45e0-8043-822b19afa29f" containerName="pull" Sep 18 06:31:53.572477 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:53.572349 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddfb5d76-3a4b-45e0-8043-822b19afa29f" containerName="pull" Sep 18 06:31:53.572477 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:53.572378 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddfb5d76-3a4b-45e0-8043-822b19afa29f" containerName="extract" Sep 18 06:31:53.572477 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:53.572414 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddfb5d76-3a4b-45e0-8043-822b19afa29f" containerName="extract" Sep 18 06:31:53.572477 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:53.572429 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddfb5d76-3a4b-45e0-8043-822b19afa29f" containerName="util" Sep 18 06:31:53.572477 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:53.572434 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddfb5d76-3a4b-45e0-8043-822b19afa29f" containerName="util" Sep 18 06:31:53.572695 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:53.572499 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddfb5d76-3a4b-45e0-8043-822b19afa29f" containerName="extract" Sep 18 06:31:53.574715 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:53.574694 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-64c86f57d4-cmwvd" Sep 18 06:31:53.578349 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:53.578314 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 18 06:31:53.578349 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:53.578319 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 18 06:31:53.578533 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:53.578319 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 18 06:31:53.578533 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:53.578387 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 18 06:31:53.578533 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:53.578324 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 18 06:31:53.578533 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:53.578320 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-pjpgd\"" Sep 18 06:31:53.586106 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:53.586083 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-64c86f57d4-cmwvd"] Sep 18 06:31:53.647997 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:53.647971 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/74420ed8-08f5-471f-9681-5558063a090c-cert\") pod \"lws-controller-manager-64c86f57d4-cmwvd\" (UID: \"74420ed8-08f5-471f-9681-5558063a090c\") " pod="openshift-lws-operator/lws-controller-manager-64c86f57d4-cmwvd" Sep 18 06:31:53.648112 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:53.648022 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/74420ed8-08f5-471f-9681-5558063a090c-manager-config\") pod \"lws-controller-manager-64c86f57d4-cmwvd\" (UID: \"74420ed8-08f5-471f-9681-5558063a090c\") " pod="openshift-lws-operator/lws-controller-manager-64c86f57d4-cmwvd" Sep 18 06:31:53.648112 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:53.648058 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6kzd\" (UniqueName: \"kubernetes.io/projected/74420ed8-08f5-471f-9681-5558063a090c-kube-api-access-q6kzd\") pod \"lws-controller-manager-64c86f57d4-cmwvd\" (UID: \"74420ed8-08f5-471f-9681-5558063a090c\") " pod="openshift-lws-operator/lws-controller-manager-64c86f57d4-cmwvd" Sep 18 06:31:53.648112 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:53.648089 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/74420ed8-08f5-471f-9681-5558063a090c-metrics-cert\") pod \"lws-controller-manager-64c86f57d4-cmwvd\" (UID: \"74420ed8-08f5-471f-9681-5558063a090c\") " pod="openshift-lws-operator/lws-controller-manager-64c86f57d4-cmwvd" Sep 18 06:31:53.748724 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:53.748691 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6kzd\" (UniqueName: \"kubernetes.io/projected/74420ed8-08f5-471f-9681-5558063a090c-kube-api-access-q6kzd\") pod \"lws-controller-manager-64c86f57d4-cmwvd\" (UID: \"74420ed8-08f5-471f-9681-5558063a090c\") " pod="openshift-lws-operator/lws-controller-manager-64c86f57d4-cmwvd" Sep 18 06:31:53.748866 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:53.748735 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/74420ed8-08f5-471f-9681-5558063a090c-metrics-cert\") pod \"lws-controller-manager-64c86f57d4-cmwvd\" (UID: \"74420ed8-08f5-471f-9681-5558063a090c\") " pod="openshift-lws-operator/lws-controller-manager-64c86f57d4-cmwvd" Sep 18 06:31:53.748866 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:53.748774 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/74420ed8-08f5-471f-9681-5558063a090c-cert\") pod \"lws-controller-manager-64c86f57d4-cmwvd\" (UID: \"74420ed8-08f5-471f-9681-5558063a090c\") " pod="openshift-lws-operator/lws-controller-manager-64c86f57d4-cmwvd" Sep 18 06:31:53.748866 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:53.748805 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/74420ed8-08f5-471f-9681-5558063a090c-manager-config\") pod \"lws-controller-manager-64c86f57d4-cmwvd\" (UID: \"74420ed8-08f5-471f-9681-5558063a090c\") " pod="openshift-lws-operator/lws-controller-manager-64c86f57d4-cmwvd" Sep 18 06:31:53.749564 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:53.749540 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/74420ed8-08f5-471f-9681-5558063a090c-manager-config\") pod \"lws-controller-manager-64c86f57d4-cmwvd\" (UID: \"74420ed8-08f5-471f-9681-5558063a090c\") " pod="openshift-lws-operator/lws-controller-manager-64c86f57d4-cmwvd" Sep 18 06:31:53.751269 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:53.751243 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/74420ed8-08f5-471f-9681-5558063a090c-metrics-cert\") pod \"lws-controller-manager-64c86f57d4-cmwvd\" (UID: \"74420ed8-08f5-471f-9681-5558063a090c\") " pod="openshift-lws-operator/lws-controller-manager-64c86f57d4-cmwvd" Sep 18 06:31:53.751353 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:53.751308 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/74420ed8-08f5-471f-9681-5558063a090c-cert\") pod \"lws-controller-manager-64c86f57d4-cmwvd\" (UID: \"74420ed8-08f5-471f-9681-5558063a090c\") " pod="openshift-lws-operator/lws-controller-manager-64c86f57d4-cmwvd" Sep 18 06:31:53.756454 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:53.756428 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6kzd\" (UniqueName: \"kubernetes.io/projected/74420ed8-08f5-471f-9681-5558063a090c-kube-api-access-q6kzd\") pod \"lws-controller-manager-64c86f57d4-cmwvd\" (UID: \"74420ed8-08f5-471f-9681-5558063a090c\") " pod="openshift-lws-operator/lws-controller-manager-64c86f57d4-cmwvd" Sep 18 06:31:53.883203 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:53.883170 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-64c86f57d4-cmwvd" Sep 18 06:31:54.149249 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:54.149225 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-64c86f57d4-cmwvd"] Sep 18 06:31:54.152830 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:31:54.152800 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74420ed8_08f5_471f_9681_5558063a090c.slice/crio-0dad4d66c55e73c582ba3886bd1a9b1a98fc7f892f33d70ebcd330653eb271f6 WatchSource:0}: Error finding container 0dad4d66c55e73c582ba3886bd1a9b1a98fc7f892f33d70ebcd330653eb271f6: Status 404 returned error can't find the container with id 0dad4d66c55e73c582ba3886bd1a9b1a98fc7f892f33d70ebcd330653eb271f6 Sep 18 06:31:54.543999 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:54.543914 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-64c86f57d4-cmwvd" event={"ID":"74420ed8-08f5-471f-9681-5558063a090c","Type":"ContainerStarted","Data":"0dad4d66c55e73c582ba3886bd1a9b1a98fc7f892f33d70ebcd330653eb271f6"} Sep 18 06:31:56.556056 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:56.556021 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-64c86f57d4-cmwvd" event={"ID":"74420ed8-08f5-471f-9681-5558063a090c","Type":"ContainerStarted","Data":"56f4a80a938dd544451563569c286d0621705865e699711ef212fea0d0fa359f"} Sep 18 06:31:56.556476 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:56.556090 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-64c86f57d4-cmwvd" Sep 18 06:31:56.572346 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:31:56.572299 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-64c86f57d4-cmwvd" podStartSLOduration=2.065399827 podStartE2EDuration="3.572286265s" totalImagesPullingTime="1.506886438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:31:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:31:56.571065338 +0000 UTC m=+578.310481529" watchObservedRunningTime="2026-09-18 06:31:56.572286265 +0000 UTC m=+578.311702439" Sep 18 06:32:02.412277 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:02.412244 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6fd575c64f-7mcbn"] Sep 18 06:32:02.414739 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:02.414719 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6fd575c64f-7mcbn" Sep 18 06:32:02.417268 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:02.417243 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 18 06:32:02.417468 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:02.417251 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 18 06:32:02.417468 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:02.417293 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-zlgzp\"" Sep 18 06:32:02.417468 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:02.417296 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 18 06:32:02.417713 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:02.417697 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 18 06:32:02.428072 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:02.428050 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6fd575c64f-7mcbn"] Sep 18 06:32:02.483332 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:02.483306 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08qrh9"] Sep 18 06:32:02.485737 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:02.485722 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08qrh9" Sep 18 06:32:02.488153 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:02.488129 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 18 06:32:02.488261 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:02.488132 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-hg9g9\"" Sep 18 06:32:02.488261 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:02.488188 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 18 06:32:02.496430 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:02.496410 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08qrh9"] Sep 18 06:32:02.518105 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:02.518078 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/44103c01-51b0-4d76-95e5-cb3b273f78ab-webhook-cert\") pod \"opendatahub-operator-controller-manager-6fd575c64f-7mcbn\" (UID: \"44103c01-51b0-4d76-95e5-cb3b273f78ab\") " pod="opendatahub/opendatahub-operator-controller-manager-6fd575c64f-7mcbn" Sep 18 06:32:02.518204 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:02.518110 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/44103c01-51b0-4d76-95e5-cb3b273f78ab-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6fd575c64f-7mcbn\" (UID: \"44103c01-51b0-4d76-95e5-cb3b273f78ab\") " pod="opendatahub/opendatahub-operator-controller-manager-6fd575c64f-7mcbn" Sep 18 06:32:02.518204 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:02.518131 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/44103c01-51b0-4d76-95e5-cb3b273f78ab-manifests\") pod \"opendatahub-operator-controller-manager-6fd575c64f-7mcbn\" (UID: \"44103c01-51b0-4d76-95e5-cb3b273f78ab\") " pod="opendatahub/opendatahub-operator-controller-manager-6fd575c64f-7mcbn" Sep 18 06:32:02.518204 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:02.518161 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss9l5\" (UniqueName: \"kubernetes.io/projected/abf2aef4-94db-4dfe-9f4b-ba2d7d8603be-kube-api-access-ss9l5\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08qrh9\" (UID: \"abf2aef4-94db-4dfe-9f4b-ba2d7d8603be\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08qrh9" Sep 18 06:32:02.518339 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:02.518216 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/abf2aef4-94db-4dfe-9f4b-ba2d7d8603be-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08qrh9\" (UID: \"abf2aef4-94db-4dfe-9f4b-ba2d7d8603be\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08qrh9" Sep 18 06:32:02.518339 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:02.518244 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdl69\" (UniqueName: \"kubernetes.io/projected/44103c01-51b0-4d76-95e5-cb3b273f78ab-kube-api-access-cdl69\") pod \"opendatahub-operator-controller-manager-6fd575c64f-7mcbn\" (UID: \"44103c01-51b0-4d76-95e5-cb3b273f78ab\") " pod="opendatahub/opendatahub-operator-controller-manager-6fd575c64f-7mcbn" Sep 18 06:32:02.518448 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:02.518433 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/abf2aef4-94db-4dfe-9f4b-ba2d7d8603be-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08qrh9\" (UID: \"abf2aef4-94db-4dfe-9f4b-ba2d7d8603be\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08qrh9" Sep 18 06:32:02.518497 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:02.518475 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/44103c01-51b0-4d76-95e5-cb3b273f78ab-tmp\") pod \"opendatahub-operator-controller-manager-6fd575c64f-7mcbn\" (UID: \"44103c01-51b0-4d76-95e5-cb3b273f78ab\") " pod="opendatahub/opendatahub-operator-controller-manager-6fd575c64f-7mcbn" Sep 18 06:32:02.619213 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:02.619182 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/abf2aef4-94db-4dfe-9f4b-ba2d7d8603be-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08qrh9\" (UID: \"abf2aef4-94db-4dfe-9f4b-ba2d7d8603be\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08qrh9" Sep 18 06:32:02.619507 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:02.619231 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdl69\" (UniqueName: \"kubernetes.io/projected/44103c01-51b0-4d76-95e5-cb3b273f78ab-kube-api-access-cdl69\") pod \"opendatahub-operator-controller-manager-6fd575c64f-7mcbn\" (UID: \"44103c01-51b0-4d76-95e5-cb3b273f78ab\") " pod="opendatahub/opendatahub-operator-controller-manager-6fd575c64f-7mcbn" Sep 18 06:32:02.619507 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:02.619253 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/abf2aef4-94db-4dfe-9f4b-ba2d7d8603be-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08qrh9\" (UID: \"abf2aef4-94db-4dfe-9f4b-ba2d7d8603be\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08qrh9" Sep 18 06:32:02.619507 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:02.619276 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/44103c01-51b0-4d76-95e5-cb3b273f78ab-tmp\") pod \"opendatahub-operator-controller-manager-6fd575c64f-7mcbn\" (UID: \"44103c01-51b0-4d76-95e5-cb3b273f78ab\") " pod="opendatahub/opendatahub-operator-controller-manager-6fd575c64f-7mcbn" Sep 18 06:32:02.619507 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:02.619303 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/44103c01-51b0-4d76-95e5-cb3b273f78ab-webhook-cert\") pod \"opendatahub-operator-controller-manager-6fd575c64f-7mcbn\" (UID: \"44103c01-51b0-4d76-95e5-cb3b273f78ab\") " pod="opendatahub/opendatahub-operator-controller-manager-6fd575c64f-7mcbn" Sep 18 06:32:02.619507 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:02.619326 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/44103c01-51b0-4d76-95e5-cb3b273f78ab-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6fd575c64f-7mcbn\" (UID: \"44103c01-51b0-4d76-95e5-cb3b273f78ab\") " pod="opendatahub/opendatahub-operator-controller-manager-6fd575c64f-7mcbn" Sep 18 06:32:02.619507 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:02.619348 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/44103c01-51b0-4d76-95e5-cb3b273f78ab-manifests\") pod \"opendatahub-operator-controller-manager-6fd575c64f-7mcbn\" (UID: \"44103c01-51b0-4d76-95e5-cb3b273f78ab\") " pod="opendatahub/opendatahub-operator-controller-manager-6fd575c64f-7mcbn" Sep 18 06:32:02.619507 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:02.619416 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss9l5\" (UniqueName: \"kubernetes.io/projected/abf2aef4-94db-4dfe-9f4b-ba2d7d8603be-kube-api-access-ss9l5\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08qrh9\" (UID: \"abf2aef4-94db-4dfe-9f4b-ba2d7d8603be\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08qrh9" Sep 18 06:32:02.619891 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:02.619599 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/abf2aef4-94db-4dfe-9f4b-ba2d7d8603be-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08qrh9\" (UID: \"abf2aef4-94db-4dfe-9f4b-ba2d7d8603be\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08qrh9" Sep 18 06:32:02.619891 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:02.619654 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/abf2aef4-94db-4dfe-9f4b-ba2d7d8603be-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08qrh9\" (UID: \"abf2aef4-94db-4dfe-9f4b-ba2d7d8603be\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08qrh9" Sep 18 06:32:02.619891 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:02.619881 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/44103c01-51b0-4d76-95e5-cb3b273f78ab-manifests\") pod \"opendatahub-operator-controller-manager-6fd575c64f-7mcbn\" (UID: \"44103c01-51b0-4d76-95e5-cb3b273f78ab\") " pod="opendatahub/opendatahub-operator-controller-manager-6fd575c64f-7mcbn" Sep 18 06:32:02.621612 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:02.621591 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/44103c01-51b0-4d76-95e5-cb3b273f78ab-tmp\") pod \"opendatahub-operator-controller-manager-6fd575c64f-7mcbn\" (UID: \"44103c01-51b0-4d76-95e5-cb3b273f78ab\") " pod="opendatahub/opendatahub-operator-controller-manager-6fd575c64f-7mcbn" Sep 18 06:32:02.621935 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:02.621915 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/44103c01-51b0-4d76-95e5-cb3b273f78ab-webhook-cert\") pod \"opendatahub-operator-controller-manager-6fd575c64f-7mcbn\" (UID: \"44103c01-51b0-4d76-95e5-cb3b273f78ab\") " pod="opendatahub/opendatahub-operator-controller-manager-6fd575c64f-7mcbn" Sep 18 06:32:02.622019 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:02.621982 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/44103c01-51b0-4d76-95e5-cb3b273f78ab-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6fd575c64f-7mcbn\" (UID: \"44103c01-51b0-4d76-95e5-cb3b273f78ab\") " pod="opendatahub/opendatahub-operator-controller-manager-6fd575c64f-7mcbn" Sep 18 06:32:02.627437 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:02.627414 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdl69\" (UniqueName: \"kubernetes.io/projected/44103c01-51b0-4d76-95e5-cb3b273f78ab-kube-api-access-cdl69\") pod \"opendatahub-operator-controller-manager-6fd575c64f-7mcbn\" (UID: \"44103c01-51b0-4d76-95e5-cb3b273f78ab\") " pod="opendatahub/opendatahub-operator-controller-manager-6fd575c64f-7mcbn" Sep 18 06:32:02.627531 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:02.627495 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss9l5\" (UniqueName: \"kubernetes.io/projected/abf2aef4-94db-4dfe-9f4b-ba2d7d8603be-kube-api-access-ss9l5\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08qrh9\" (UID: \"abf2aef4-94db-4dfe-9f4b-ba2d7d8603be\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08qrh9" Sep 18 06:32:02.726570 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:02.726496 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6fd575c64f-7mcbn" Sep 18 06:32:02.794869 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:02.794833 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08qrh9" Sep 18 06:32:03.008466 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:03.008440 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6fd575c64f-7mcbn"] Sep 18 06:32:03.010807 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:32:03.010780 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44103c01_51b0_4d76_95e5_cb3b273f78ab.slice/crio-8649086c60d6bc4fa1cf39183f01525409e144a0565d92e649e1cc2f6cba3828 WatchSource:0}: Error finding container 8649086c60d6bc4fa1cf39183f01525409e144a0565d92e649e1cc2f6cba3828: Status 404 returned error can't find the container with id 8649086c60d6bc4fa1cf39183f01525409e144a0565d92e649e1cc2f6cba3828 Sep 18 06:32:03.066418 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:03.066351 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08qrh9"] Sep 18 06:32:03.068536 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:32:03.068507 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabf2aef4_94db_4dfe_9f4b_ba2d7d8603be.slice/crio-35e18ee5ed3eaf738d504a586a0fffe9ee81052bf85631c95effe21ed5c77943 WatchSource:0}: Error finding container 35e18ee5ed3eaf738d504a586a0fffe9ee81052bf85631c95effe21ed5c77943: Status 404 returned error can't find the container with id 35e18ee5ed3eaf738d504a586a0fffe9ee81052bf85631c95effe21ed5c77943 Sep 18 06:32:03.599427 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:03.597956 2567 generic.go:356] "Generic (PLEG): container finished" podID="abf2aef4-94db-4dfe-9f4b-ba2d7d8603be" containerID="3d69b9c33e6a7144309aca3c3b5cc6e80fdc754c95396ff450c91f5746f88e07" exitCode=0 Sep 18 06:32:03.599427 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:03.598094 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08qrh9" event={"ID":"abf2aef4-94db-4dfe-9f4b-ba2d7d8603be","Type":"ContainerDied","Data":"3d69b9c33e6a7144309aca3c3b5cc6e80fdc754c95396ff450c91f5746f88e07"} Sep 18 06:32:03.599427 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:03.598147 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08qrh9" event={"ID":"abf2aef4-94db-4dfe-9f4b-ba2d7d8603be","Type":"ContainerStarted","Data":"35e18ee5ed3eaf738d504a586a0fffe9ee81052bf85631c95effe21ed5c77943"} Sep 18 06:32:03.602572 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:03.602452 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6fd575c64f-7mcbn" event={"ID":"44103c01-51b0-4d76-95e5-cb3b273f78ab","Type":"ContainerStarted","Data":"8649086c60d6bc4fa1cf39183f01525409e144a0565d92e649e1cc2f6cba3828"} Sep 18 06:32:04.610977 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:04.610948 2567 generic.go:356] "Generic (PLEG): container finished" podID="abf2aef4-94db-4dfe-9f4b-ba2d7d8603be" containerID="5fa4b8aec8d2f6c87f712eada46c89fa1fc4efc94196332d4f85405d809dd65a" exitCode=0 Sep 18 06:32:04.611426 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:04.611019 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08qrh9" event={"ID":"abf2aef4-94db-4dfe-9f4b-ba2d7d8603be","Type":"ContainerDied","Data":"5fa4b8aec8d2f6c87f712eada46c89fa1fc4efc94196332d4f85405d809dd65a"} Sep 18 06:32:06.622205 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:06.622173 2567 generic.go:356] "Generic (PLEG): container finished" podID="abf2aef4-94db-4dfe-9f4b-ba2d7d8603be" containerID="9b6d2e1220570a6aada3b805cafd006a127bf2cfd2a80a3d4ae598cfd29cb98b" exitCode=0 Sep 18 06:32:06.622594 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:06.622254 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08qrh9" event={"ID":"abf2aef4-94db-4dfe-9f4b-ba2d7d8603be","Type":"ContainerDied","Data":"9b6d2e1220570a6aada3b805cafd006a127bf2cfd2a80a3d4ae598cfd29cb98b"} Sep 18 06:32:06.623787 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:06.623764 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6fd575c64f-7mcbn" event={"ID":"44103c01-51b0-4d76-95e5-cb3b273f78ab","Type":"ContainerStarted","Data":"4de19ab26d949f3665755700c59bb7c50d4e944c7d1b031929399435eeb9c33e"} Sep 18 06:32:07.562050 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:07.561966 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-64c86f57d4-cmwvd" Sep 18 06:32:07.631027 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:07.630993 2567 generic.go:356] "Generic (PLEG): container finished" podID="44103c01-51b0-4d76-95e5-cb3b273f78ab" containerID="4de19ab26d949f3665755700c59bb7c50d4e944c7d1b031929399435eeb9c33e" exitCode=0 Sep 18 06:32:07.631395 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:07.631076 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6fd575c64f-7mcbn" event={"ID":"44103c01-51b0-4d76-95e5-cb3b273f78ab","Type":"ContainerDied","Data":"4de19ab26d949f3665755700c59bb7c50d4e944c7d1b031929399435eeb9c33e"} Sep 18 06:32:07.754187 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:07.754166 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08qrh9" Sep 18 06:32:07.867779 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:07.867739 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ss9l5\" (UniqueName: \"kubernetes.io/projected/abf2aef4-94db-4dfe-9f4b-ba2d7d8603be-kube-api-access-ss9l5\") pod \"abf2aef4-94db-4dfe-9f4b-ba2d7d8603be\" (UID: \"abf2aef4-94db-4dfe-9f4b-ba2d7d8603be\") " Sep 18 06:32:07.867779 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:07.867785 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/abf2aef4-94db-4dfe-9f4b-ba2d7d8603be-util\") pod \"abf2aef4-94db-4dfe-9f4b-ba2d7d8603be\" (UID: \"abf2aef4-94db-4dfe-9f4b-ba2d7d8603be\") " Sep 18 06:32:07.867995 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:07.867804 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/abf2aef4-94db-4dfe-9f4b-ba2d7d8603be-bundle\") pod \"abf2aef4-94db-4dfe-9f4b-ba2d7d8603be\" (UID: \"abf2aef4-94db-4dfe-9f4b-ba2d7d8603be\") " Sep 18 06:32:07.868492 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:07.868470 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abf2aef4-94db-4dfe-9f4b-ba2d7d8603be-bundle" (OuterVolumeSpecName: "bundle") pod "abf2aef4-94db-4dfe-9f4b-ba2d7d8603be" (UID: "abf2aef4-94db-4dfe-9f4b-ba2d7d8603be"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:32:07.869862 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:07.869840 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abf2aef4-94db-4dfe-9f4b-ba2d7d8603be-kube-api-access-ss9l5" (OuterVolumeSpecName: "kube-api-access-ss9l5") pod "abf2aef4-94db-4dfe-9f4b-ba2d7d8603be" (UID: "abf2aef4-94db-4dfe-9f4b-ba2d7d8603be"). InnerVolumeSpecName "kube-api-access-ss9l5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 06:32:07.872514 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:07.872477 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abf2aef4-94db-4dfe-9f4b-ba2d7d8603be-util" (OuterVolumeSpecName: "util") pod "abf2aef4-94db-4dfe-9f4b-ba2d7d8603be" (UID: "abf2aef4-94db-4dfe-9f4b-ba2d7d8603be"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:32:07.968828 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:07.968794 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ss9l5\" (UniqueName: \"kubernetes.io/projected/abf2aef4-94db-4dfe-9f4b-ba2d7d8603be-kube-api-access-ss9l5\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:32:07.968828 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:07.968826 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/abf2aef4-94db-4dfe-9f4b-ba2d7d8603be-util\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:32:07.968828 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:07.968836 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/abf2aef4-94db-4dfe-9f4b-ba2d7d8603be-bundle\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:32:08.638038 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:08.638007 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08qrh9" Sep 18 06:32:08.638454 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:08.638033 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08qrh9" event={"ID":"abf2aef4-94db-4dfe-9f4b-ba2d7d8603be","Type":"ContainerDied","Data":"35e18ee5ed3eaf738d504a586a0fffe9ee81052bf85631c95effe21ed5c77943"} Sep 18 06:32:08.638454 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:08.638086 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35e18ee5ed3eaf738d504a586a0fffe9ee81052bf85631c95effe21ed5c77943" Sep 18 06:32:08.640358 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:08.640335 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6fd575c64f-7mcbn" event={"ID":"44103c01-51b0-4d76-95e5-cb3b273f78ab","Type":"ContainerStarted","Data":"2e08379a29f9956cda24efffe227748acf73c7bcbb5c55b05407d52d443d9349"} Sep 18 06:32:08.640711 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:08.640696 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6fd575c64f-7mcbn" Sep 18 06:32:08.662408 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:08.662343 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6fd575c64f-7mcbn" podStartSLOduration=2.226836741 podStartE2EDuration="6.6623261s" totalImagesPullingTime="3.389674264s" totalInitContainerRuntime="1.045815095s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:32:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:32:08.660001437 +0000 UTC m=+590.399417632" watchObservedRunningTime="2026-09-18 06:32:08.6623261 +0000 UTC m=+590.401742278" Sep 18 06:32:19.647703 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:19.647674 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6fd575c64f-7mcbn" Sep 18 06:32:20.357361 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:20.357332 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjr4h_eb059951-e263-40da-9155-4fd0cf242820/ovn-acl-logging/0.log" Sep 18 06:32:20.357858 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:20.357836 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjr4h_eb059951-e263-40da-9155-4fd0cf242820/ovn-acl-logging/0.log" Sep 18 06:32:28.388607 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:28.388574 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg"] Sep 18 06:32:28.389973 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:28.388876 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abf2aef4-94db-4dfe-9f4b-ba2d7d8603be" containerName="util" Sep 18 06:32:28.389973 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:28.388887 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="abf2aef4-94db-4dfe-9f4b-ba2d7d8603be" containerName="util" Sep 18 06:32:28.389973 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:28.388903 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abf2aef4-94db-4dfe-9f4b-ba2d7d8603be" containerName="extract" Sep 18 06:32:28.389973 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:28.388909 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="abf2aef4-94db-4dfe-9f4b-ba2d7d8603be" containerName="extract" Sep 18 06:32:28.389973 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:28.388920 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abf2aef4-94db-4dfe-9f4b-ba2d7d8603be" containerName="pull" Sep 18 06:32:28.389973 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:28.388925 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="abf2aef4-94db-4dfe-9f4b-ba2d7d8603be" containerName="pull" Sep 18 06:32:28.389973 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:28.388993 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="abf2aef4-94db-4dfe-9f4b-ba2d7d8603be" containerName="extract" Sep 18 06:32:28.391051 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:28.391035 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg" Sep 18 06:32:28.395304 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:28.395280 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-djh74\"" Sep 18 06:32:28.395703 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:28.395681 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 18 06:32:28.405289 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:28.405264 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg"] Sep 18 06:32:28.524589 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:28.524561 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a4b02c0-6cda-40d5-a37a-05f0ba6ea4b4-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg\" (UID: \"2a4b02c0-6cda-40d5-a37a-05f0ba6ea4b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg" Sep 18 06:32:28.524726 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:28.524594 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a4b02c0-6cda-40d5-a37a-05f0ba6ea4b4-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg\" (UID: \"2a4b02c0-6cda-40d5-a37a-05f0ba6ea4b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg" Sep 18 06:32:28.524726 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:28.524611 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a4b02c0-6cda-40d5-a37a-05f0ba6ea4b4-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg\" (UID: \"2a4b02c0-6cda-40d5-a37a-05f0ba6ea4b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg" Sep 18 06:32:28.524726 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:28.524629 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a4b02c0-6cda-40d5-a37a-05f0ba6ea4b4-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg\" (UID: \"2a4b02c0-6cda-40d5-a37a-05f0ba6ea4b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg" Sep 18 06:32:28.524726 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:28.524702 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a4b02c0-6cda-40d5-a37a-05f0ba6ea4b4-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg\" (UID: \"2a4b02c0-6cda-40d5-a37a-05f0ba6ea4b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg" Sep 18 06:32:28.524881 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:28.524733 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b74m2\" (UniqueName: \"kubernetes.io/projected/2a4b02c0-6cda-40d5-a37a-05f0ba6ea4b4-kube-api-access-b74m2\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg\" (UID: \"2a4b02c0-6cda-40d5-a37a-05f0ba6ea4b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg" Sep 18 06:32:28.524881 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:28.524813 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a4b02c0-6cda-40d5-a37a-05f0ba6ea4b4-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg\" (UID: \"2a4b02c0-6cda-40d5-a37a-05f0ba6ea4b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg" Sep 18 06:32:28.524881 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:28.524828 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a4b02c0-6cda-40d5-a37a-05f0ba6ea4b4-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg\" (UID: \"2a4b02c0-6cda-40d5-a37a-05f0ba6ea4b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg" Sep 18 06:32:28.524881 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:28.524864 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a4b02c0-6cda-40d5-a37a-05f0ba6ea4b4-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg\" (UID: \"2a4b02c0-6cda-40d5-a37a-05f0ba6ea4b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg" Sep 18 06:32:28.625900 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:28.625865 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a4b02c0-6cda-40d5-a37a-05f0ba6ea4b4-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg\" (UID: \"2a4b02c0-6cda-40d5-a37a-05f0ba6ea4b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg" Sep 18 06:32:28.626042 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:28.625903 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a4b02c0-6cda-40d5-a37a-05f0ba6ea4b4-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg\" (UID: \"2a4b02c0-6cda-40d5-a37a-05f0ba6ea4b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg" Sep 18 06:32:28.626042 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:28.625931 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a4b02c0-6cda-40d5-a37a-05f0ba6ea4b4-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg\" (UID: \"2a4b02c0-6cda-40d5-a37a-05f0ba6ea4b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg" Sep 18 06:32:28.626042 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:28.625963 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a4b02c0-6cda-40d5-a37a-05f0ba6ea4b4-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg\" (UID: \"2a4b02c0-6cda-40d5-a37a-05f0ba6ea4b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg" Sep 18 06:32:28.626042 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:28.625987 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b74m2\" (UniqueName: \"kubernetes.io/projected/2a4b02c0-6cda-40d5-a37a-05f0ba6ea4b4-kube-api-access-b74m2\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg\" (UID: \"2a4b02c0-6cda-40d5-a37a-05f0ba6ea4b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg" Sep 18 06:32:28.626258 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:28.626067 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a4b02c0-6cda-40d5-a37a-05f0ba6ea4b4-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg\" (UID: \"2a4b02c0-6cda-40d5-a37a-05f0ba6ea4b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg" Sep 18 06:32:28.626258 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:28.626092 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a4b02c0-6cda-40d5-a37a-05f0ba6ea4b4-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg\" (UID: \"2a4b02c0-6cda-40d5-a37a-05f0ba6ea4b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg" Sep 18 06:32:28.626258 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:28.626137 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a4b02c0-6cda-40d5-a37a-05f0ba6ea4b4-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg\" (UID: \"2a4b02c0-6cda-40d5-a37a-05f0ba6ea4b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg" Sep 18 06:32:28.626258 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:28.626171 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a4b02c0-6cda-40d5-a37a-05f0ba6ea4b4-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg\" (UID: \"2a4b02c0-6cda-40d5-a37a-05f0ba6ea4b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg" Sep 18 06:32:28.626480 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:28.626256 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a4b02c0-6cda-40d5-a37a-05f0ba6ea4b4-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg\" (UID: \"2a4b02c0-6cda-40d5-a37a-05f0ba6ea4b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg" Sep 18 06:32:28.626536 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:28.626490 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a4b02c0-6cda-40d5-a37a-05f0ba6ea4b4-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg\" (UID: \"2a4b02c0-6cda-40d5-a37a-05f0ba6ea4b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg" Sep 18 06:32:28.626594 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:28.626563 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a4b02c0-6cda-40d5-a37a-05f0ba6ea4b4-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg\" (UID: \"2a4b02c0-6cda-40d5-a37a-05f0ba6ea4b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg" Sep 18 06:32:28.626714 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:28.626692 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a4b02c0-6cda-40d5-a37a-05f0ba6ea4b4-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg\" (UID: \"2a4b02c0-6cda-40d5-a37a-05f0ba6ea4b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg" Sep 18 06:32:28.626819 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:28.626802 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a4b02c0-6cda-40d5-a37a-05f0ba6ea4b4-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg\" (UID: \"2a4b02c0-6cda-40d5-a37a-05f0ba6ea4b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg" Sep 18 06:32:28.628450 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:28.628428 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a4b02c0-6cda-40d5-a37a-05f0ba6ea4b4-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg\" (UID: \"2a4b02c0-6cda-40d5-a37a-05f0ba6ea4b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg" Sep 18 06:32:28.628548 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:28.628534 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a4b02c0-6cda-40d5-a37a-05f0ba6ea4b4-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg\" (UID: \"2a4b02c0-6cda-40d5-a37a-05f0ba6ea4b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg" Sep 18 06:32:28.633498 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:28.633480 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a4b02c0-6cda-40d5-a37a-05f0ba6ea4b4-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg\" (UID: \"2a4b02c0-6cda-40d5-a37a-05f0ba6ea4b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg" Sep 18 06:32:28.633595 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:28.633551 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b74m2\" (UniqueName: \"kubernetes.io/projected/2a4b02c0-6cda-40d5-a37a-05f0ba6ea4b4-kube-api-access-b74m2\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg\" (UID: \"2a4b02c0-6cda-40d5-a37a-05f0ba6ea4b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg" Sep 18 06:32:28.702739 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:28.702690 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg" Sep 18 06:32:28.974774 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:28.974697 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg"] Sep 18 06:32:28.978614 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:32:28.978585 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a4b02c0_6cda_40d5_a37a_05f0ba6ea4b4.slice/crio-c3e6bc51c1affe964af2c4dc97dc3b0f7b2ed847d010f1d8ed6a8d51604e7720 WatchSource:0}: Error finding container c3e6bc51c1affe964af2c4dc97dc3b0f7b2ed847d010f1d8ed6a8d51604e7720: Status 404 returned error can't find the container with id c3e6bc51c1affe964af2c4dc97dc3b0f7b2ed847d010f1d8ed6a8d51604e7720 Sep 18 06:32:29.743865 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:29.743831 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg" event={"ID":"2a4b02c0-6cda-40d5-a37a-05f0ba6ea4b4","Type":"ContainerStarted","Data":"c3e6bc51c1affe964af2c4dc97dc3b0f7b2ed847d010f1d8ed6a8d51604e7720"} Sep 18 06:32:31.783407 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:31.783348 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:32:31.783772 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:31.783454 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:32:31.783772 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:31.783498 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:32:32.606660 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:32.606621 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-qgwmc"] Sep 18 06:32:32.609603 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:32.609586 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-qgwmc" Sep 18 06:32:32.612145 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:32.612125 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Sep 18 06:32:32.612682 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:32.612662 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-8hvh2\"" Sep 18 06:32:32.617728 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:32.617707 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-qgwmc"] Sep 18 06:32:32.760498 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:32.760463 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg" event={"ID":"2a4b02c0-6cda-40d5-a37a-05f0ba6ea4b4","Type":"ContainerStarted","Data":"a786eb4edfe97f4e10f6853ebc3861907abd1e76e88f87f36071d90d565f63e0"} Sep 18 06:32:32.763701 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:32.763671 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kq6m\" (UniqueName: \"kubernetes.io/projected/61e31de6-574c-43c5-a839-9dd1d9067587-kube-api-access-9kq6m\") pod \"odh-model-controller-7998c67dc4-qgwmc\" (UID: \"61e31de6-574c-43c5-a839-9dd1d9067587\") " pod="opendatahub/odh-model-controller-7998c67dc4-qgwmc" Sep 18 06:32:32.763783 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:32.763718 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/61e31de6-574c-43c5-a839-9dd1d9067587-cert\") pod \"odh-model-controller-7998c67dc4-qgwmc\" (UID: \"61e31de6-574c-43c5-a839-9dd1d9067587\") " pod="opendatahub/odh-model-controller-7998c67dc4-qgwmc" Sep 18 06:32:32.779666 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:32.779616 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg" podStartSLOduration=1.976957723 podStartE2EDuration="4.779601178s" totalImagesPullingTime="2.802643455s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:32:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:32:32.778543653 +0000 UTC m=+614.517959854" watchObservedRunningTime="2026-09-18 06:32:32.779601178 +0000 UTC m=+614.519017753" Sep 18 06:32:32.864807 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:32.864708 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kq6m\" (UniqueName: \"kubernetes.io/projected/61e31de6-574c-43c5-a839-9dd1d9067587-kube-api-access-9kq6m\") pod \"odh-model-controller-7998c67dc4-qgwmc\" (UID: \"61e31de6-574c-43c5-a839-9dd1d9067587\") " pod="opendatahub/odh-model-controller-7998c67dc4-qgwmc" Sep 18 06:32:32.864807 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:32.864761 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/61e31de6-574c-43c5-a839-9dd1d9067587-cert\") pod \"odh-model-controller-7998c67dc4-qgwmc\" (UID: \"61e31de6-574c-43c5-a839-9dd1d9067587\") " pod="opendatahub/odh-model-controller-7998c67dc4-qgwmc" Sep 18 06:32:32.867703 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:32.867676 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/61e31de6-574c-43c5-a839-9dd1d9067587-cert\") pod \"odh-model-controller-7998c67dc4-qgwmc\" (UID: \"61e31de6-574c-43c5-a839-9dd1d9067587\") " pod="opendatahub/odh-model-controller-7998c67dc4-qgwmc" Sep 18 06:32:32.872332 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:32.872300 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kq6m\" (UniqueName: \"kubernetes.io/projected/61e31de6-574c-43c5-a839-9dd1d9067587-kube-api-access-9kq6m\") pod \"odh-model-controller-7998c67dc4-qgwmc\" (UID: \"61e31de6-574c-43c5-a839-9dd1d9067587\") " pod="opendatahub/odh-model-controller-7998c67dc4-qgwmc" Sep 18 06:32:32.920277 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:32.920253 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-qgwmc" Sep 18 06:32:33.197460 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:33.197282 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-qgwmc"] Sep 18 06:32:33.199525 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:32:33.199500 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61e31de6_574c_43c5_a839_9dd1d9067587.slice/crio-6b20e16c71bf06d2fb5583b8ad4fa0cd1ebf4a5120dd3b0a86fd48a03d1e77cd WatchSource:0}: Error finding container 6b20e16c71bf06d2fb5583b8ad4fa0cd1ebf4a5120dd3b0a86fd48a03d1e77cd: Status 404 returned error can't find the container with id 6b20e16c71bf06d2fb5583b8ad4fa0cd1ebf4a5120dd3b0a86fd48a03d1e77cd Sep 18 06:32:33.703694 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:33.703661 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg" Sep 18 06:32:33.708329 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:33.708302 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg" Sep 18 06:32:33.767990 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:33.767950 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-qgwmc" event={"ID":"61e31de6-574c-43c5-a839-9dd1d9067587","Type":"ContainerStarted","Data":"6b20e16c71bf06d2fb5583b8ad4fa0cd1ebf4a5120dd3b0a86fd48a03d1e77cd"} Sep 18 06:32:33.768255 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:33.768236 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg" Sep 18 06:32:33.769122 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:33.769100 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg" Sep 18 06:32:36.786708 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:36.786676 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-qgwmc" event={"ID":"61e31de6-574c-43c5-a839-9dd1d9067587","Type":"ContainerStarted","Data":"c77de917c44869ebffd9181fe891663ded7a4873a1d1cbeaa77b740ce884b414"} Sep 18 06:32:36.787132 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:36.786814 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-qgwmc" Sep 18 06:32:36.803119 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:36.803076 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-qgwmc" podStartSLOduration=2.026685062 podStartE2EDuration="4.803062107s" totalImagesPullingTime="2.776377045s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:32:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:32:36.801600526 +0000 UTC m=+618.541016720" watchObservedRunningTime="2026-09-18 06:32:36.803062107 +0000 UTC m=+618.542478280" Sep 18 06:32:39.722066 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:39.722031 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-rzl5d"] Sep 18 06:32:39.725803 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:39.725780 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rzl5d" Sep 18 06:32:39.731090 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:39.731066 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-jrfxp\"" Sep 18 06:32:39.732029 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:39.731554 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 18 06:32:39.732957 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:39.731593 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 18 06:32:39.733059 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:39.731682 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 18 06:32:39.738344 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:39.738314 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-rzl5d"] Sep 18 06:32:39.831822 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:39.831774 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/87337e45-0211-4fb0-bf69-7b19c2934525-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-rzl5d\" (UID: \"87337e45-0211-4fb0-bf69-7b19c2934525\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rzl5d" Sep 18 06:32:39.832025 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:39.831827 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94l6h\" (UniqueName: \"kubernetes.io/projected/87337e45-0211-4fb0-bf69-7b19c2934525-kube-api-access-94l6h\") pod \"istiod-openshift-gateway-5677ccb49b-rzl5d\" (UID: \"87337e45-0211-4fb0-bf69-7b19c2934525\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rzl5d" Sep 18 06:32:39.832025 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:39.831872 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/87337e45-0211-4fb0-bf69-7b19c2934525-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-rzl5d\" (UID: \"87337e45-0211-4fb0-bf69-7b19c2934525\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rzl5d" Sep 18 06:32:39.832025 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:39.831937 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/87337e45-0211-4fb0-bf69-7b19c2934525-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-rzl5d\" (UID: \"87337e45-0211-4fb0-bf69-7b19c2934525\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rzl5d" Sep 18 06:32:39.832025 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:39.832009 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87337e45-0211-4fb0-bf69-7b19c2934525-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-rzl5d\" (UID: \"87337e45-0211-4fb0-bf69-7b19c2934525\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rzl5d" Sep 18 06:32:39.832264 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:39.832085 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/87337e45-0211-4fb0-bf69-7b19c2934525-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-rzl5d\" (UID: \"87337e45-0211-4fb0-bf69-7b19c2934525\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rzl5d" Sep 18 06:32:39.832264 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:39.832203 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/87337e45-0211-4fb0-bf69-7b19c2934525-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-rzl5d\" (UID: \"87337e45-0211-4fb0-bf69-7b19c2934525\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rzl5d" Sep 18 06:32:39.933615 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:39.933581 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87337e45-0211-4fb0-bf69-7b19c2934525-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-rzl5d\" (UID: \"87337e45-0211-4fb0-bf69-7b19c2934525\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rzl5d" Sep 18 06:32:39.933615 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:39.933618 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/87337e45-0211-4fb0-bf69-7b19c2934525-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-rzl5d\" (UID: \"87337e45-0211-4fb0-bf69-7b19c2934525\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rzl5d" Sep 18 06:32:39.933854 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:39.933660 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/87337e45-0211-4fb0-bf69-7b19c2934525-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-rzl5d\" (UID: \"87337e45-0211-4fb0-bf69-7b19c2934525\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rzl5d" Sep 18 06:32:39.933854 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:39.933690 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/87337e45-0211-4fb0-bf69-7b19c2934525-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-rzl5d\" (UID: \"87337e45-0211-4fb0-bf69-7b19c2934525\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rzl5d" Sep 18 06:32:39.933854 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:39.933711 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94l6h\" (UniqueName: \"kubernetes.io/projected/87337e45-0211-4fb0-bf69-7b19c2934525-kube-api-access-94l6h\") pod \"istiod-openshift-gateway-5677ccb49b-rzl5d\" (UID: \"87337e45-0211-4fb0-bf69-7b19c2934525\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rzl5d" Sep 18 06:32:39.933854 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:39.933735 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/87337e45-0211-4fb0-bf69-7b19c2934525-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-rzl5d\" (UID: \"87337e45-0211-4fb0-bf69-7b19c2934525\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rzl5d" Sep 18 06:32:39.933854 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:39.933784 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/87337e45-0211-4fb0-bf69-7b19c2934525-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-rzl5d\" (UID: \"87337e45-0211-4fb0-bf69-7b19c2934525\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rzl5d" Sep 18 06:32:39.934677 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:39.934642 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/87337e45-0211-4fb0-bf69-7b19c2934525-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-rzl5d\" (UID: \"87337e45-0211-4fb0-bf69-7b19c2934525\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rzl5d" Sep 18 06:32:39.936009 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:39.935975 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/87337e45-0211-4fb0-bf69-7b19c2934525-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-rzl5d\" (UID: \"87337e45-0211-4fb0-bf69-7b19c2934525\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rzl5d" Sep 18 06:32:39.936135 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:39.936101 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/87337e45-0211-4fb0-bf69-7b19c2934525-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-rzl5d\" (UID: \"87337e45-0211-4fb0-bf69-7b19c2934525\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rzl5d" Sep 18 06:32:39.936190 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:39.936144 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/87337e45-0211-4fb0-bf69-7b19c2934525-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-rzl5d\" (UID: \"87337e45-0211-4fb0-bf69-7b19c2934525\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rzl5d" Sep 18 06:32:39.936190 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:39.936161 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/87337e45-0211-4fb0-bf69-7b19c2934525-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-rzl5d\" (UID: \"87337e45-0211-4fb0-bf69-7b19c2934525\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rzl5d" Sep 18 06:32:39.946251 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:39.946229 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87337e45-0211-4fb0-bf69-7b19c2934525-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-rzl5d\" (UID: \"87337e45-0211-4fb0-bf69-7b19c2934525\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rzl5d" Sep 18 06:32:39.946649 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:39.946631 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94l6h\" (UniqueName: \"kubernetes.io/projected/87337e45-0211-4fb0-bf69-7b19c2934525-kube-api-access-94l6h\") pod \"istiod-openshift-gateway-5677ccb49b-rzl5d\" (UID: \"87337e45-0211-4fb0-bf69-7b19c2934525\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rzl5d" Sep 18 06:32:40.041354 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:40.041284 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rzl5d" Sep 18 06:32:40.348462 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:40.348430 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-rzl5d"] Sep 18 06:32:40.350121 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:32:40.350091 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87337e45_0211_4fb0_bf69_7b19c2934525.slice/crio-28baeb401998a6cc8165a146db13917ebd61d03de3d94c48f6d322cef6297980 WatchSource:0}: Error finding container 28baeb401998a6cc8165a146db13917ebd61d03de3d94c48f6d322cef6297980: Status 404 returned error can't find the container with id 28baeb401998a6cc8165a146db13917ebd61d03de3d94c48f6d322cef6297980 Sep 18 06:32:40.352329 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:40.352312 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 18 06:32:40.811065 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:40.810959 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rzl5d" event={"ID":"87337e45-0211-4fb0-bf69-7b19c2934525","Type":"ContainerStarted","Data":"28baeb401998a6cc8165a146db13917ebd61d03de3d94c48f6d322cef6297980"} Sep 18 06:32:42.626709 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:42.626675 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:32:42.627070 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:42.626757 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:32:42.831276 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:42.831236 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rzl5d" event={"ID":"87337e45-0211-4fb0-bf69-7b19c2934525","Type":"ContainerStarted","Data":"dd324f9f16f1e83537f8178cd6bd159e7ed90d94c3520b95ee42c8170f7d1d8d"} Sep 18 06:32:42.831447 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:42.831347 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rzl5d" Sep 18 06:32:42.855502 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:42.855456 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rzl5d" podStartSLOduration=1.581521366 podStartE2EDuration="3.855442846s" totalImagesPullingTime="2.27392148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:32:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:32:42.853745544 +0000 UTC m=+624.593161739" watchObservedRunningTime="2026-09-18 06:32:42.855442846 +0000 UTC m=+624.594859019" Sep 18 06:32:43.838516 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:43.838481 2567 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-rzl5d container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 18 06:32:43.838949 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:43.838546 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rzl5d" podUID="87337e45-0211-4fb0-bf69-7b19c2934525" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 18 06:32:46.837241 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:46.837209 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rzl5d" Sep 18 06:32:47.794765 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:32:47.794739 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-qgwmc" Sep 18 06:33:37.025171 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:37.025138 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76999z"] Sep 18 06:33:37.028131 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:37.028110 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76999z" Sep 18 06:33:37.034384 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:37.031287 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 18 06:33:37.034384 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:37.031623 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 18 06:33:37.036764 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:37.036742 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-hg9g9\"" Sep 18 06:33:37.039829 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:37.039809 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76999z"] Sep 18 06:33:37.127154 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:37.127127 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clxq8x"] Sep 18 06:33:37.129700 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:37.129684 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clxq8x" Sep 18 06:33:37.137895 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:37.137873 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clxq8x"] Sep 18 06:33:37.186304 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:37.186277 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f8d4ebf0-2824-406f-b912-d1f49ade15ac-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76999z\" (UID: \"f8d4ebf0-2824-406f-b912-d1f49ade15ac\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76999z" Sep 18 06:33:37.186432 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:37.186326 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-848bn\" (UniqueName: \"kubernetes.io/projected/f8d4ebf0-2824-406f-b912-d1f49ade15ac-kube-api-access-848bn\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76999z\" (UID: \"f8d4ebf0-2824-406f-b912-d1f49ade15ac\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76999z" Sep 18 06:33:37.186432 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:37.186416 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f8d4ebf0-2824-406f-b912-d1f49ade15ac-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76999z\" (UID: \"f8d4ebf0-2824-406f-b912-d1f49ade15ac\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76999z" Sep 18 06:33:37.225902 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:37.225872 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69lbr2"] Sep 18 06:33:37.228301 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:37.228284 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69lbr2" Sep 18 06:33:37.237582 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:37.237561 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69lbr2"] Sep 18 06:33:37.287297 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:37.287240 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f8d4ebf0-2824-406f-b912-d1f49ade15ac-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76999z\" (UID: \"f8d4ebf0-2824-406f-b912-d1f49ade15ac\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76999z" Sep 18 06:33:37.287421 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:37.287399 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67l46\" (UniqueName: \"kubernetes.io/projected/b167109b-1317-42c8-9816-14becde53cb7-kube-api-access-67l46\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clxq8x\" (UID: \"b167109b-1317-42c8-9816-14becde53cb7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clxq8x" Sep 18 06:33:37.287469 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:37.287431 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b167109b-1317-42c8-9816-14becde53cb7-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clxq8x\" (UID: \"b167109b-1317-42c8-9816-14becde53cb7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clxq8x" Sep 18 06:33:37.287469 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:37.287456 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f8d4ebf0-2824-406f-b912-d1f49ade15ac-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76999z\" (UID: \"f8d4ebf0-2824-406f-b912-d1f49ade15ac\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76999z" Sep 18 06:33:37.287540 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:37.287523 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-848bn\" (UniqueName: \"kubernetes.io/projected/f8d4ebf0-2824-406f-b912-d1f49ade15ac-kube-api-access-848bn\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76999z\" (UID: \"f8d4ebf0-2824-406f-b912-d1f49ade15ac\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76999z" Sep 18 06:33:37.287584 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:37.287550 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b167109b-1317-42c8-9816-14becde53cb7-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clxq8x\" (UID: \"b167109b-1317-42c8-9816-14becde53cb7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clxq8x" Sep 18 06:33:37.287584 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:37.287569 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f8d4ebf0-2824-406f-b912-d1f49ade15ac-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76999z\" (UID: \"f8d4ebf0-2824-406f-b912-d1f49ade15ac\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76999z" Sep 18 06:33:37.287745 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:37.287730 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f8d4ebf0-2824-406f-b912-d1f49ade15ac-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76999z\" (UID: \"f8d4ebf0-2824-406f-b912-d1f49ade15ac\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76999z" Sep 18 06:33:37.295298 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:37.295278 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-848bn\" (UniqueName: \"kubernetes.io/projected/f8d4ebf0-2824-406f-b912-d1f49ade15ac-kube-api-access-848bn\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76999z\" (UID: \"f8d4ebf0-2824-406f-b912-d1f49ade15ac\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76999z" Sep 18 06:33:37.329599 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:37.329571 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5vvv"] Sep 18 06:33:37.332092 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:37.332074 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5vvv" Sep 18 06:33:37.340633 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:37.340613 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5vvv"] Sep 18 06:33:37.344105 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:37.344088 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76999z" Sep 18 06:33:37.388850 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:37.388817 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67l46\" (UniqueName: \"kubernetes.io/projected/b167109b-1317-42c8-9816-14becde53cb7-kube-api-access-67l46\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clxq8x\" (UID: \"b167109b-1317-42c8-9816-14becde53cb7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clxq8x" Sep 18 06:33:37.388986 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:37.388861 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b167109b-1317-42c8-9816-14becde53cb7-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clxq8x\" (UID: \"b167109b-1317-42c8-9816-14becde53cb7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clxq8x" Sep 18 06:33:37.388986 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:37.388973 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fb8f7d5f-40af-46b7-b5d8-5e8d5d26ff7e-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69lbr2\" (UID: \"fb8f7d5f-40af-46b7-b5d8-5e8d5d26ff7e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69lbr2" Sep 18 06:33:37.389103 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:37.389028 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b167109b-1317-42c8-9816-14becde53cb7-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clxq8x\" (UID: \"b167109b-1317-42c8-9816-14becde53cb7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clxq8x" Sep 18 06:33:37.389103 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:37.389055 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fb8f7d5f-40af-46b7-b5d8-5e8d5d26ff7e-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69lbr2\" (UID: \"fb8f7d5f-40af-46b7-b5d8-5e8d5d26ff7e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69lbr2" Sep 18 06:33:37.389206 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:37.389099 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm2sq\" (UniqueName: \"kubernetes.io/projected/fb8f7d5f-40af-46b7-b5d8-5e8d5d26ff7e-kube-api-access-dm2sq\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69lbr2\" (UID: \"fb8f7d5f-40af-46b7-b5d8-5e8d5d26ff7e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69lbr2" Sep 18 06:33:37.389417 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:37.389396 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b167109b-1317-42c8-9816-14becde53cb7-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clxq8x\" (UID: \"b167109b-1317-42c8-9816-14becde53cb7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clxq8x" Sep 18 06:33:37.389558 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:37.389464 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b167109b-1317-42c8-9816-14becde53cb7-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clxq8x\" (UID: \"b167109b-1317-42c8-9816-14becde53cb7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clxq8x" Sep 18 06:33:37.398486 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:37.398462 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67l46\" (UniqueName: \"kubernetes.io/projected/b167109b-1317-42c8-9816-14becde53cb7-kube-api-access-67l46\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clxq8x\" (UID: \"b167109b-1317-42c8-9816-14becde53cb7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clxq8x" Sep 18 06:33:37.439531 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:37.439499 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clxq8x" Sep 18 06:33:37.491595 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:37.491529 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fb8f7d5f-40af-46b7-b5d8-5e8d5d26ff7e-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69lbr2\" (UID: \"fb8f7d5f-40af-46b7-b5d8-5e8d5d26ff7e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69lbr2" Sep 18 06:33:37.491721 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:37.491604 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fb8f7d5f-40af-46b7-b5d8-5e8d5d26ff7e-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69lbr2\" (UID: \"fb8f7d5f-40af-46b7-b5d8-5e8d5d26ff7e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69lbr2" Sep 18 06:33:37.491721 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:37.491639 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e21e964-831b-487e-9703-96bae58a0dd2-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5vvv\" (UID: \"2e21e964-831b-487e-9703-96bae58a0dd2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5vvv" Sep 18 06:33:37.491721 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:37.491673 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm2sq\" (UniqueName: \"kubernetes.io/projected/fb8f7d5f-40af-46b7-b5d8-5e8d5d26ff7e-kube-api-access-dm2sq\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69lbr2\" (UID: \"fb8f7d5f-40af-46b7-b5d8-5e8d5d26ff7e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69lbr2" Sep 18 06:33:37.491887 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:37.491748 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4j9v\" (UniqueName: \"kubernetes.io/projected/2e21e964-831b-487e-9703-96bae58a0dd2-kube-api-access-k4j9v\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5vvv\" (UID: \"2e21e964-831b-487e-9703-96bae58a0dd2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5vvv" Sep 18 06:33:37.491887 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:37.491785 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e21e964-831b-487e-9703-96bae58a0dd2-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5vvv\" (UID: \"2e21e964-831b-487e-9703-96bae58a0dd2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5vvv" Sep 18 06:33:37.492299 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:37.492269 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fb8f7d5f-40af-46b7-b5d8-5e8d5d26ff7e-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69lbr2\" (UID: \"fb8f7d5f-40af-46b7-b5d8-5e8d5d26ff7e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69lbr2" Sep 18 06:33:37.492600 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:37.492578 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fb8f7d5f-40af-46b7-b5d8-5e8d5d26ff7e-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69lbr2\" (UID: \"fb8f7d5f-40af-46b7-b5d8-5e8d5d26ff7e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69lbr2" Sep 18 06:33:37.503769 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:37.503491 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm2sq\" (UniqueName: \"kubernetes.io/projected/fb8f7d5f-40af-46b7-b5d8-5e8d5d26ff7e-kube-api-access-dm2sq\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69lbr2\" (UID: \"fb8f7d5f-40af-46b7-b5d8-5e8d5d26ff7e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69lbr2" Sep 18 06:33:37.538016 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:37.537952 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69lbr2" Sep 18 06:33:37.594905 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:37.593489 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e21e964-831b-487e-9703-96bae58a0dd2-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5vvv\" (UID: \"2e21e964-831b-487e-9703-96bae58a0dd2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5vvv" Sep 18 06:33:37.594905 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:37.593603 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4j9v\" (UniqueName: \"kubernetes.io/projected/2e21e964-831b-487e-9703-96bae58a0dd2-kube-api-access-k4j9v\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5vvv\" (UID: \"2e21e964-831b-487e-9703-96bae58a0dd2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5vvv" Sep 18 06:33:37.594905 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:37.593641 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e21e964-831b-487e-9703-96bae58a0dd2-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5vvv\" (UID: \"2e21e964-831b-487e-9703-96bae58a0dd2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5vvv" Sep 18 06:33:37.594905 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:37.594232 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e21e964-831b-487e-9703-96bae58a0dd2-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5vvv\" (UID: \"2e21e964-831b-487e-9703-96bae58a0dd2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5vvv" Sep 18 06:33:37.594905 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:37.594538 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e21e964-831b-487e-9703-96bae58a0dd2-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5vvv\" (UID: \"2e21e964-831b-487e-9703-96bae58a0dd2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5vvv" Sep 18 06:33:37.603286 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:37.603265 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4j9v\" (UniqueName: \"kubernetes.io/projected/2e21e964-831b-487e-9703-96bae58a0dd2-kube-api-access-k4j9v\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5vvv\" (UID: \"2e21e964-831b-487e-9703-96bae58a0dd2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5vvv" Sep 18 06:33:37.617785 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:37.617764 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76999z"] Sep 18 06:33:37.619468 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:33:37.619445 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8d4ebf0_2824_406f_b912_d1f49ade15ac.slice/crio-5d41d1a596b1cb7f72d9fef8f417f5db3facf3f3fbaa446ee8ec29b356b363af WatchSource:0}: Error finding container 5d41d1a596b1cb7f72d9fef8f417f5db3facf3f3fbaa446ee8ec29b356b363af: Status 404 returned error can't find the container with id 5d41d1a596b1cb7f72d9fef8f417f5db3facf3f3fbaa446ee8ec29b356b363af Sep 18 06:33:37.642069 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:37.642048 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5vvv" Sep 18 06:33:37.718066 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:37.717994 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clxq8x"] Sep 18 06:33:37.720734 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:33:37.720690 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb167109b_1317_42c8_9816_14becde53cb7.slice/crio-7a6bb770a398925fc7e479ac1ed4f3ea626c182a7a02c9685e17b160b2cf4458 WatchSource:0}: Error finding container 7a6bb770a398925fc7e479ac1ed4f3ea626c182a7a02c9685e17b160b2cf4458: Status 404 returned error can't find the container with id 7a6bb770a398925fc7e479ac1ed4f3ea626c182a7a02c9685e17b160b2cf4458 Sep 18 06:33:37.809048 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:37.809025 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69lbr2"] Sep 18 06:33:37.810233 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:33:37.810204 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb8f7d5f_40af_46b7_b5d8_5e8d5d26ff7e.slice/crio-d7f29e39134665fca60ea8b4aeabde33d2f0dd70aaf4cbbde32223f277d2a411 WatchSource:0}: Error finding container d7f29e39134665fca60ea8b4aeabde33d2f0dd70aaf4cbbde32223f277d2a411: Status 404 returned error can't find the container with id d7f29e39134665fca60ea8b4aeabde33d2f0dd70aaf4cbbde32223f277d2a411 Sep 18 06:33:37.914006 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:37.913984 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5vvv"] Sep 18 06:33:37.979233 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:33:37.979208 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e21e964_831b_487e_9703_96bae58a0dd2.slice/crio-5724d5c4378919a99c39fde59431fd972150e9d60c7016a20889d57cc8436437 WatchSource:0}: Error finding container 5724d5c4378919a99c39fde59431fd972150e9d60c7016a20889d57cc8436437: Status 404 returned error can't find the container with id 5724d5c4378919a99c39fde59431fd972150e9d60c7016a20889d57cc8436437 Sep 18 06:33:38.105342 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:38.105313 2567 generic.go:356] "Generic (PLEG): container finished" podID="fb8f7d5f-40af-46b7-b5d8-5e8d5d26ff7e" containerID="e244a6c299a7af83590341b122a415c43672295655835defcc23cd93b719993a" exitCode=0 Sep 18 06:33:38.105738 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:38.105398 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69lbr2" event={"ID":"fb8f7d5f-40af-46b7-b5d8-5e8d5d26ff7e","Type":"ContainerDied","Data":"e244a6c299a7af83590341b122a415c43672295655835defcc23cd93b719993a"} Sep 18 06:33:38.105738 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:38.105499 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69lbr2" event={"ID":"fb8f7d5f-40af-46b7-b5d8-5e8d5d26ff7e","Type":"ContainerStarted","Data":"d7f29e39134665fca60ea8b4aeabde33d2f0dd70aaf4cbbde32223f277d2a411"} Sep 18 06:33:38.107268 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:38.107242 2567 generic.go:356] "Generic (PLEG): container finished" podID="b167109b-1317-42c8-9816-14becde53cb7" containerID="4339f4edcb1377f4b7a61d138bec7bfc7c8842e5cb4f9cf679d64e520e4912a1" exitCode=0 Sep 18 06:33:38.107407 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:38.107279 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clxq8x" event={"ID":"b167109b-1317-42c8-9816-14becde53cb7","Type":"ContainerDied","Data":"4339f4edcb1377f4b7a61d138bec7bfc7c8842e5cb4f9cf679d64e520e4912a1"} Sep 18 06:33:38.107407 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:38.107332 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clxq8x" event={"ID":"b167109b-1317-42c8-9816-14becde53cb7","Type":"ContainerStarted","Data":"7a6bb770a398925fc7e479ac1ed4f3ea626c182a7a02c9685e17b160b2cf4458"} Sep 18 06:33:38.113271 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:38.113251 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5vvv" event={"ID":"2e21e964-831b-487e-9703-96bae58a0dd2","Type":"ContainerStarted","Data":"45c12681c0581a23ea7b328802df6dbe8695b664c7c86fbb81fdc9610250ef0d"} Sep 18 06:33:38.113359 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:38.113304 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5vvv" event={"ID":"2e21e964-831b-487e-9703-96bae58a0dd2","Type":"ContainerStarted","Data":"5724d5c4378919a99c39fde59431fd972150e9d60c7016a20889d57cc8436437"} Sep 18 06:33:38.115357 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:38.115335 2567 generic.go:356] "Generic (PLEG): container finished" podID="f8d4ebf0-2824-406f-b912-d1f49ade15ac" containerID="41fc305035267a97018b434272c7c8f58536a35b8711e5b3da1f4f63ad5e33d3" exitCode=0 Sep 18 06:33:38.115467 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:38.115425 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76999z" event={"ID":"f8d4ebf0-2824-406f-b912-d1f49ade15ac","Type":"ContainerDied","Data":"41fc305035267a97018b434272c7c8f58536a35b8711e5b3da1f4f63ad5e33d3"} Sep 18 06:33:38.115514 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:38.115476 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76999z" event={"ID":"f8d4ebf0-2824-406f-b912-d1f49ade15ac","Type":"ContainerStarted","Data":"5d41d1a596b1cb7f72d9fef8f417f5db3facf3f3fbaa446ee8ec29b356b363af"} Sep 18 06:33:39.122717 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:39.122684 2567 generic.go:356] "Generic (PLEG): container finished" podID="2e21e964-831b-487e-9703-96bae58a0dd2" containerID="45c12681c0581a23ea7b328802df6dbe8695b664c7c86fbb81fdc9610250ef0d" exitCode=0 Sep 18 06:33:39.123000 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:39.122765 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5vvv" event={"ID":"2e21e964-831b-487e-9703-96bae58a0dd2","Type":"ContainerDied","Data":"45c12681c0581a23ea7b328802df6dbe8695b664c7c86fbb81fdc9610250ef0d"} Sep 18 06:33:40.131062 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:40.131021 2567 generic.go:356] "Generic (PLEG): container finished" podID="b167109b-1317-42c8-9816-14becde53cb7" containerID="e2f973bb4e6d471885f34815c60b18d0fb518e187dc863a4ec99b0be719052d4" exitCode=0 Sep 18 06:33:40.131562 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:40.131099 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clxq8x" event={"ID":"b167109b-1317-42c8-9816-14becde53cb7","Type":"ContainerDied","Data":"e2f973bb4e6d471885f34815c60b18d0fb518e187dc863a4ec99b0be719052d4"} Sep 18 06:33:40.133129 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:40.133105 2567 generic.go:356] "Generic (PLEG): container finished" podID="2e21e964-831b-487e-9703-96bae58a0dd2" containerID="eeea8b9427062a8af89bb98b60750be66e411540b08f583be08a911c6d346f10" exitCode=0 Sep 18 06:33:40.133235 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:40.133173 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5vvv" event={"ID":"2e21e964-831b-487e-9703-96bae58a0dd2","Type":"ContainerDied","Data":"eeea8b9427062a8af89bb98b60750be66e411540b08f583be08a911c6d346f10"} Sep 18 06:33:40.135220 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:40.135194 2567 generic.go:356] "Generic (PLEG): container finished" podID="f8d4ebf0-2824-406f-b912-d1f49ade15ac" containerID="8decb79242998d7ef3b137130b45fae1a976d584fb7ffdf2ca2f8eb96ec4a1ea" exitCode=0 Sep 18 06:33:40.135324 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:40.135223 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76999z" event={"ID":"f8d4ebf0-2824-406f-b912-d1f49ade15ac","Type":"ContainerDied","Data":"8decb79242998d7ef3b137130b45fae1a976d584fb7ffdf2ca2f8eb96ec4a1ea"} Sep 18 06:33:40.137323 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:40.137308 2567 generic.go:356] "Generic (PLEG): container finished" podID="fb8f7d5f-40af-46b7-b5d8-5e8d5d26ff7e" containerID="bb8580689a82dd3ab4940854cb29eacf99cfdfec68e8ee0cb2966123e5b419ec" exitCode=0 Sep 18 06:33:40.137477 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:40.137412 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69lbr2" event={"ID":"fb8f7d5f-40af-46b7-b5d8-5e8d5d26ff7e","Type":"ContainerDied","Data":"bb8580689a82dd3ab4940854cb29eacf99cfdfec68e8ee0cb2966123e5b419ec"} Sep 18 06:33:41.144484 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:41.144453 2567 generic.go:356] "Generic (PLEG): container finished" podID="b167109b-1317-42c8-9816-14becde53cb7" containerID="36583a8098bc560aef7969ced29b311c0de7754c5bf5b3449797c289f5a2da8a" exitCode=0 Sep 18 06:33:41.144870 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:41.144538 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clxq8x" event={"ID":"b167109b-1317-42c8-9816-14becde53cb7","Type":"ContainerDied","Data":"36583a8098bc560aef7969ced29b311c0de7754c5bf5b3449797c289f5a2da8a"} Sep 18 06:33:41.146641 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:41.146614 2567 generic.go:356] "Generic (PLEG): container finished" podID="2e21e964-831b-487e-9703-96bae58a0dd2" containerID="8dc15b7266a5896936316a072d29316c56e6408c454e16dd9610a1517842930b" exitCode=0 Sep 18 06:33:41.146788 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:41.146658 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5vvv" event={"ID":"2e21e964-831b-487e-9703-96bae58a0dd2","Type":"ContainerDied","Data":"8dc15b7266a5896936316a072d29316c56e6408c454e16dd9610a1517842930b"} Sep 18 06:33:41.148911 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:41.148891 2567 generic.go:356] "Generic (PLEG): container finished" podID="f8d4ebf0-2824-406f-b912-d1f49ade15ac" containerID="e54f61f8828e0cb6faecdd0e82e224072bc48fc9e8ec0286507cd26973e9a924" exitCode=0 Sep 18 06:33:41.149020 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:41.148965 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76999z" event={"ID":"f8d4ebf0-2824-406f-b912-d1f49ade15ac","Type":"ContainerDied","Data":"e54f61f8828e0cb6faecdd0e82e224072bc48fc9e8ec0286507cd26973e9a924"} Sep 18 06:33:41.150844 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:41.150824 2567 generic.go:356] "Generic (PLEG): container finished" podID="fb8f7d5f-40af-46b7-b5d8-5e8d5d26ff7e" containerID="c553f022f9a6381e4b02dfaecb308456a018383786fdf037e756d80e89a770cb" exitCode=0 Sep 18 06:33:41.150953 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:41.150899 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69lbr2" event={"ID":"fb8f7d5f-40af-46b7-b5d8-5e8d5d26ff7e","Type":"ContainerDied","Data":"c553f022f9a6381e4b02dfaecb308456a018383786fdf037e756d80e89a770cb"} Sep 18 06:33:42.293606 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:42.293574 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69lbr2" Sep 18 06:33:42.348943 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:42.348923 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clxq8x" Sep 18 06:33:42.352054 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:42.352033 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76999z" Sep 18 06:33:42.355432 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:42.355413 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5vvv" Sep 18 06:33:42.436744 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:42.436659 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f8d4ebf0-2824-406f-b912-d1f49ade15ac-util\") pod \"f8d4ebf0-2824-406f-b912-d1f49ade15ac\" (UID: \"f8d4ebf0-2824-406f-b912-d1f49ade15ac\") " Sep 18 06:33:42.436744 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:42.436690 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k4j9v\" (UniqueName: \"kubernetes.io/projected/2e21e964-831b-487e-9703-96bae58a0dd2-kube-api-access-k4j9v\") pod \"2e21e964-831b-487e-9703-96bae58a0dd2\" (UID: \"2e21e964-831b-487e-9703-96bae58a0dd2\") " Sep 18 06:33:42.436744 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:42.436737 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fb8f7d5f-40af-46b7-b5d8-5e8d5d26ff7e-util\") pod \"fb8f7d5f-40af-46b7-b5d8-5e8d5d26ff7e\" (UID: \"fb8f7d5f-40af-46b7-b5d8-5e8d5d26ff7e\") " Sep 18 06:33:42.436988 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:42.436793 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-848bn\" (UniqueName: \"kubernetes.io/projected/f8d4ebf0-2824-406f-b912-d1f49ade15ac-kube-api-access-848bn\") pod \"f8d4ebf0-2824-406f-b912-d1f49ade15ac\" (UID: \"f8d4ebf0-2824-406f-b912-d1f49ade15ac\") " Sep 18 06:33:42.436988 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:42.436840 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fb8f7d5f-40af-46b7-b5d8-5e8d5d26ff7e-bundle\") pod \"fb8f7d5f-40af-46b7-b5d8-5e8d5d26ff7e\" (UID: \"fb8f7d5f-40af-46b7-b5d8-5e8d5d26ff7e\") " Sep 18 06:33:42.437108 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:42.437038 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-67l46\" (UniqueName: \"kubernetes.io/projected/b167109b-1317-42c8-9816-14becde53cb7-kube-api-access-67l46\") pod \"b167109b-1317-42c8-9816-14becde53cb7\" (UID: \"b167109b-1317-42c8-9816-14becde53cb7\") " Sep 18 06:33:42.437163 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:42.437131 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b167109b-1317-42c8-9816-14becde53cb7-bundle\") pod \"b167109b-1317-42c8-9816-14becde53cb7\" (UID: \"b167109b-1317-42c8-9816-14becde53cb7\") " Sep 18 06:33:42.437218 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:42.437162 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dm2sq\" (UniqueName: \"kubernetes.io/projected/fb8f7d5f-40af-46b7-b5d8-5e8d5d26ff7e-kube-api-access-dm2sq\") pod \"fb8f7d5f-40af-46b7-b5d8-5e8d5d26ff7e\" (UID: \"fb8f7d5f-40af-46b7-b5d8-5e8d5d26ff7e\") " Sep 18 06:33:42.437273 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:42.437213 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f8d4ebf0-2824-406f-b912-d1f49ade15ac-bundle\") pod \"f8d4ebf0-2824-406f-b912-d1f49ade15ac\" (UID: \"f8d4ebf0-2824-406f-b912-d1f49ade15ac\") " Sep 18 06:33:42.437273 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:42.437246 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b167109b-1317-42c8-9816-14becde53cb7-util\") pod \"b167109b-1317-42c8-9816-14becde53cb7\" (UID: \"b167109b-1317-42c8-9816-14becde53cb7\") " Sep 18 06:33:42.437399 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:42.437292 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb8f7d5f-40af-46b7-b5d8-5e8d5d26ff7e-bundle" (OuterVolumeSpecName: "bundle") pod "fb8f7d5f-40af-46b7-b5d8-5e8d5d26ff7e" (UID: "fb8f7d5f-40af-46b7-b5d8-5e8d5d26ff7e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:33:42.437464 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:42.437299 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e21e964-831b-487e-9703-96bae58a0dd2-bundle\") pod \"2e21e964-831b-487e-9703-96bae58a0dd2\" (UID: \"2e21e964-831b-487e-9703-96bae58a0dd2\") " Sep 18 06:33:42.437531 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:42.437477 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e21e964-831b-487e-9703-96bae58a0dd2-util\") pod \"2e21e964-831b-487e-9703-96bae58a0dd2\" (UID: \"2e21e964-831b-487e-9703-96bae58a0dd2\") " Sep 18 06:33:42.437847 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:42.437808 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fb8f7d5f-40af-46b7-b5d8-5e8d5d26ff7e-bundle\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:33:42.438557 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:42.437746 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b167109b-1317-42c8-9816-14becde53cb7-bundle" (OuterVolumeSpecName: "bundle") pod "b167109b-1317-42c8-9816-14becde53cb7" (UID: "b167109b-1317-42c8-9816-14becde53cb7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:33:42.438557 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:42.438171 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8d4ebf0-2824-406f-b912-d1f49ade15ac-bundle" (OuterVolumeSpecName: "bundle") pod "f8d4ebf0-2824-406f-b912-d1f49ade15ac" (UID: "f8d4ebf0-2824-406f-b912-d1f49ade15ac"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:33:42.438557 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:42.438505 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e21e964-831b-487e-9703-96bae58a0dd2-bundle" (OuterVolumeSpecName: "bundle") pod "2e21e964-831b-487e-9703-96bae58a0dd2" (UID: "2e21e964-831b-487e-9703-96bae58a0dd2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:33:42.439810 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:42.439773 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8d4ebf0-2824-406f-b912-d1f49ade15ac-kube-api-access-848bn" (OuterVolumeSpecName: "kube-api-access-848bn") pod "f8d4ebf0-2824-406f-b912-d1f49ade15ac" (UID: "f8d4ebf0-2824-406f-b912-d1f49ade15ac"). InnerVolumeSpecName "kube-api-access-848bn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 06:33:42.440145 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:42.440120 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb8f7d5f-40af-46b7-b5d8-5e8d5d26ff7e-kube-api-access-dm2sq" (OuterVolumeSpecName: "kube-api-access-dm2sq") pod "fb8f7d5f-40af-46b7-b5d8-5e8d5d26ff7e" (UID: "fb8f7d5f-40af-46b7-b5d8-5e8d5d26ff7e"). InnerVolumeSpecName "kube-api-access-dm2sq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 06:33:42.440291 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:42.440267 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e21e964-831b-487e-9703-96bae58a0dd2-kube-api-access-k4j9v" (OuterVolumeSpecName: "kube-api-access-k4j9v") pod "2e21e964-831b-487e-9703-96bae58a0dd2" (UID: "2e21e964-831b-487e-9703-96bae58a0dd2"). InnerVolumeSpecName "kube-api-access-k4j9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 06:33:42.440740 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:42.440706 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b167109b-1317-42c8-9816-14becde53cb7-kube-api-access-67l46" (OuterVolumeSpecName: "kube-api-access-67l46") pod "b167109b-1317-42c8-9816-14becde53cb7" (UID: "b167109b-1317-42c8-9816-14becde53cb7"). InnerVolumeSpecName "kube-api-access-67l46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 06:33:42.446245 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:42.446218 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8d4ebf0-2824-406f-b912-d1f49ade15ac-util" (OuterVolumeSpecName: "util") pod "f8d4ebf0-2824-406f-b912-d1f49ade15ac" (UID: "f8d4ebf0-2824-406f-b912-d1f49ade15ac"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:33:42.446928 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:42.446906 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e21e964-831b-487e-9703-96bae58a0dd2-util" (OuterVolumeSpecName: "util") pod "2e21e964-831b-487e-9703-96bae58a0dd2" (UID: "2e21e964-831b-487e-9703-96bae58a0dd2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:33:42.447455 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:42.447429 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b167109b-1317-42c8-9816-14becde53cb7-util" (OuterVolumeSpecName: "util") pod "b167109b-1317-42c8-9816-14becde53cb7" (UID: "b167109b-1317-42c8-9816-14becde53cb7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:33:42.448565 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:42.448542 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb8f7d5f-40af-46b7-b5d8-5e8d5d26ff7e-util" (OuterVolumeSpecName: "util") pod "fb8f7d5f-40af-46b7-b5d8-5e8d5d26ff7e" (UID: "fb8f7d5f-40af-46b7-b5d8-5e8d5d26ff7e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:33:42.539103 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:42.539077 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-848bn\" (UniqueName: \"kubernetes.io/projected/f8d4ebf0-2824-406f-b912-d1f49ade15ac-kube-api-access-848bn\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:33:42.539103 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:42.539099 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67l46\" (UniqueName: \"kubernetes.io/projected/b167109b-1317-42c8-9816-14becde53cb7-kube-api-access-67l46\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:33:42.539243 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:42.539109 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b167109b-1317-42c8-9816-14becde53cb7-bundle\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:33:42.539243 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:42.539119 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm2sq\" (UniqueName: \"kubernetes.io/projected/fb8f7d5f-40af-46b7-b5d8-5e8d5d26ff7e-kube-api-access-dm2sq\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:33:42.539243 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:42.539129 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f8d4ebf0-2824-406f-b912-d1f49ade15ac-bundle\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:33:42.539243 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:42.539137 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b167109b-1317-42c8-9816-14becde53cb7-util\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:33:42.539243 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:42.539145 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e21e964-831b-487e-9703-96bae58a0dd2-bundle\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:33:42.539243 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:42.539153 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e21e964-831b-487e-9703-96bae58a0dd2-util\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:33:42.539243 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:42.539160 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f8d4ebf0-2824-406f-b912-d1f49ade15ac-util\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:33:42.539243 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:42.539167 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4j9v\" (UniqueName: \"kubernetes.io/projected/2e21e964-831b-487e-9703-96bae58a0dd2-kube-api-access-k4j9v\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:33:42.539243 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:42.539175 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fb8f7d5f-40af-46b7-b5d8-5e8d5d26ff7e-util\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:33:43.162801 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:43.162776 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clxq8x" Sep 18 06:33:43.162976 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:43.162777 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clxq8x" event={"ID":"b167109b-1317-42c8-9816-14becde53cb7","Type":"ContainerDied","Data":"7a6bb770a398925fc7e479ac1ed4f3ea626c182a7a02c9685e17b160b2cf4458"} Sep 18 06:33:43.162976 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:43.162890 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a6bb770a398925fc7e479ac1ed4f3ea626c182a7a02c9685e17b160b2cf4458" Sep 18 06:33:43.164825 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:43.164806 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5vvv" Sep 18 06:33:43.164939 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:43.164803 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at5vvv" event={"ID":"2e21e964-831b-487e-9703-96bae58a0dd2","Type":"ContainerDied","Data":"5724d5c4378919a99c39fde59431fd972150e9d60c7016a20889d57cc8436437"} Sep 18 06:33:43.164939 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:43.164912 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5724d5c4378919a99c39fde59431fd972150e9d60c7016a20889d57cc8436437" Sep 18 06:33:43.167081 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:43.167021 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76999z" event={"ID":"f8d4ebf0-2824-406f-b912-d1f49ade15ac","Type":"ContainerDied","Data":"5d41d1a596b1cb7f72d9fef8f417f5db3facf3f3fbaa446ee8ec29b356b363af"} Sep 18 06:33:43.167081 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:43.167034 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76999z" Sep 18 06:33:43.167081 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:43.167059 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d41d1a596b1cb7f72d9fef8f417f5db3facf3f3fbaa446ee8ec29b356b363af" Sep 18 06:33:43.169230 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:43.169208 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69lbr2" event={"ID":"fb8f7d5f-40af-46b7-b5d8-5e8d5d26ff7e","Type":"ContainerDied","Data":"d7f29e39134665fca60ea8b4aeabde33d2f0dd70aaf4cbbde32223f277d2a411"} Sep 18 06:33:43.169316 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:43.169246 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7f29e39134665fca60ea8b4aeabde33d2f0dd70aaf4cbbde32223f277d2a411" Sep 18 06:33:43.169316 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:33:43.169253 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69lbr2" Sep 18 06:36:10.445846 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:10.445814 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-jbzm5"] Sep 18 06:36:10.446243 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:10.446117 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b167109b-1317-42c8-9816-14becde53cb7" containerName="util" Sep 18 06:36:10.446243 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:10.446127 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b167109b-1317-42c8-9816-14becde53cb7" containerName="util" Sep 18 06:36:10.446243 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:10.446135 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b167109b-1317-42c8-9816-14becde53cb7" containerName="pull" Sep 18 06:36:10.446243 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:10.446141 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b167109b-1317-42c8-9816-14becde53cb7" containerName="pull" Sep 18 06:36:10.446243 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:10.446149 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb8f7d5f-40af-46b7-b5d8-5e8d5d26ff7e" containerName="pull" Sep 18 06:36:10.446243 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:10.446155 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb8f7d5f-40af-46b7-b5d8-5e8d5d26ff7e" containerName="pull" Sep 18 06:36:10.446243 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:10.446161 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8d4ebf0-2824-406f-b912-d1f49ade15ac" containerName="extract" Sep 18 06:36:10.446243 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:10.446166 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8d4ebf0-2824-406f-b912-d1f49ade15ac" containerName="extract" Sep 18 06:36:10.446243 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:10.446174 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8d4ebf0-2824-406f-b912-d1f49ade15ac" containerName="pull" Sep 18 06:36:10.446243 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:10.446179 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8d4ebf0-2824-406f-b912-d1f49ade15ac" containerName="pull" Sep 18 06:36:10.446243 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:10.446187 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e21e964-831b-487e-9703-96bae58a0dd2" containerName="pull" Sep 18 06:36:10.446243 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:10.446193 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e21e964-831b-487e-9703-96bae58a0dd2" containerName="pull" Sep 18 06:36:10.446243 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:10.446199 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b167109b-1317-42c8-9816-14becde53cb7" containerName="extract" Sep 18 06:36:10.446243 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:10.446205 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b167109b-1317-42c8-9816-14becde53cb7" containerName="extract" Sep 18 06:36:10.446243 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:10.446211 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb8f7d5f-40af-46b7-b5d8-5e8d5d26ff7e" containerName="extract" Sep 18 06:36:10.446243 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:10.446216 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb8f7d5f-40af-46b7-b5d8-5e8d5d26ff7e" containerName="extract" Sep 18 06:36:10.446243 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:10.446227 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e21e964-831b-487e-9703-96bae58a0dd2" containerName="util" Sep 18 06:36:10.446243 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:10.446232 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e21e964-831b-487e-9703-96bae58a0dd2" containerName="util" Sep 18 06:36:10.446243 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:10.446239 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8d4ebf0-2824-406f-b912-d1f49ade15ac" containerName="util" Sep 18 06:36:10.446243 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:10.446244 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8d4ebf0-2824-406f-b912-d1f49ade15ac" containerName="util" Sep 18 06:36:10.446243 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:10.446252 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb8f7d5f-40af-46b7-b5d8-5e8d5d26ff7e" containerName="util" Sep 18 06:36:10.446868 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:10.446257 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb8f7d5f-40af-46b7-b5d8-5e8d5d26ff7e" containerName="util" Sep 18 06:36:10.446868 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:10.446262 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e21e964-831b-487e-9703-96bae58a0dd2" containerName="extract" Sep 18 06:36:10.446868 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:10.446266 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e21e964-831b-487e-9703-96bae58a0dd2" containerName="extract" Sep 18 06:36:10.446868 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:10.446319 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8d4ebf0-2824-406f-b912-d1f49ade15ac" containerName="extract" Sep 18 06:36:10.446868 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:10.446328 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb8f7d5f-40af-46b7-b5d8-5e8d5d26ff7e" containerName="extract" Sep 18 06:36:10.446868 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:10.446334 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="b167109b-1317-42c8-9816-14becde53cb7" containerName="extract" Sep 18 06:36:10.446868 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:10.446338 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e21e964-831b-487e-9703-96bae58a0dd2" containerName="extract" Sep 18 06:36:10.448319 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:10.448299 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-jbzm5" Sep 18 06:36:10.451241 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:10.451218 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 18 06:36:10.451572 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:10.451546 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-5kv4r\"" Sep 18 06:36:10.453783 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:10.453756 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-jbzm5"] Sep 18 06:36:10.603330 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:10.603297 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qrct\" (UniqueName: \"kubernetes.io/projected/96a61b61-7de1-428e-9fb6-99d5534fb7db-kube-api-access-2qrct\") pod \"payload-processing-5b54b7fbb7-jbzm5\" (UID: \"96a61b61-7de1-428e-9fb6-99d5534fb7db\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-jbzm5" Sep 18 06:36:10.603330 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:10.603332 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96a61b61-7de1-428e-9fb6-99d5534fb7db-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-jbzm5\" (UID: \"96a61b61-7de1-428e-9fb6-99d5534fb7db\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-jbzm5" Sep 18 06:36:10.703885 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:10.703797 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qrct\" (UniqueName: \"kubernetes.io/projected/96a61b61-7de1-428e-9fb6-99d5534fb7db-kube-api-access-2qrct\") pod \"payload-processing-5b54b7fbb7-jbzm5\" (UID: \"96a61b61-7de1-428e-9fb6-99d5534fb7db\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-jbzm5" Sep 18 06:36:10.703885 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:10.703843 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96a61b61-7de1-428e-9fb6-99d5534fb7db-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-jbzm5\" (UID: \"96a61b61-7de1-428e-9fb6-99d5534fb7db\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-jbzm5" Sep 18 06:36:10.704590 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:10.704564 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96a61b61-7de1-428e-9fb6-99d5534fb7db-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-jbzm5\" (UID: \"96a61b61-7de1-428e-9fb6-99d5534fb7db\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-jbzm5" Sep 18 06:36:10.711024 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:10.711005 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qrct\" (UniqueName: \"kubernetes.io/projected/96a61b61-7de1-428e-9fb6-99d5534fb7db-kube-api-access-2qrct\") pod \"payload-processing-5b54b7fbb7-jbzm5\" (UID: \"96a61b61-7de1-428e-9fb6-99d5534fb7db\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-jbzm5" Sep 18 06:36:10.760195 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:10.760168 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-jbzm5" Sep 18 06:36:11.029008 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:11.028982 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-jbzm5"] Sep 18 06:36:11.030568 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:36:11.030536 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96a61b61_7de1_428e_9fb6_99d5534fb7db.slice/crio-fd7149b25f31d746926c29ef5af16db21c2c2fa87fa68d5184b56de61687457d WatchSource:0}: Error finding container fd7149b25f31d746926c29ef5af16db21c2c2fa87fa68d5184b56de61687457d: Status 404 returned error can't find the container with id fd7149b25f31d746926c29ef5af16db21c2c2fa87fa68d5184b56de61687457d Sep 18 06:36:11.924494 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:11.924460 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-jbzm5" event={"ID":"96a61b61-7de1-428e-9fb6-99d5534fb7db","Type":"ContainerStarted","Data":"fd7149b25f31d746926c29ef5af16db21c2c2fa87fa68d5184b56de61687457d"} Sep 18 06:36:12.932483 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:12.932448 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-jbzm5" event={"ID":"96a61b61-7de1-428e-9fb6-99d5534fb7db","Type":"ContainerStarted","Data":"d99ac163912bed600ad988d864800fd122edcee49698deebe8140c5cac6521f9"} Sep 18 06:36:12.932845 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:12.932605 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-jbzm5" Sep 18 06:36:12.947693 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:12.947653 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-jbzm5" podStartSLOduration=1.6575258320000001 podStartE2EDuration="2.947640014s" totalImagesPullingTime="1.290114182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:36:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:36:12.946690649 +0000 UTC m=+834.686106838" watchObservedRunningTime="2026-09-18 06:36:12.947640014 +0000 UTC m=+834.687056396" Sep 18 06:36:23.939473 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:23.939439 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-jbzm5" Sep 18 06:36:35.184186 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:35.184155 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hdknc"] Sep 18 06:36:35.187779 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:35.187764 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hdknc" Sep 18 06:36:35.191333 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:35.191313 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 18 06:36:35.191333 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:35.191324 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 18 06:36:35.191535 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:35.191342 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-wzjhw\"" Sep 18 06:36:35.191535 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:35.191443 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Sep 18 06:36:35.196957 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:35.196935 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hdknc"] Sep 18 06:36:35.284196 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:35.284162 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d43a82f2-a138-4b8e-917f-bf0b2ab4aa5c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hdknc\" (UID: \"d43a82f2-a138-4b8e-917f-bf0b2ab4aa5c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hdknc" Sep 18 06:36:35.284196 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:35.284199 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d43a82f2-a138-4b8e-917f-bf0b2ab4aa5c-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hdknc\" (UID: \"d43a82f2-a138-4b8e-917f-bf0b2ab4aa5c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hdknc" Sep 18 06:36:35.284416 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:35.284235 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvlxb\" (UniqueName: \"kubernetes.io/projected/d43a82f2-a138-4b8e-917f-bf0b2ab4aa5c-kube-api-access-pvlxb\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hdknc\" (UID: \"d43a82f2-a138-4b8e-917f-bf0b2ab4aa5c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hdknc" Sep 18 06:36:35.284416 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:35.284265 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d43a82f2-a138-4b8e-917f-bf0b2ab4aa5c-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hdknc\" (UID: \"d43a82f2-a138-4b8e-917f-bf0b2ab4aa5c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hdknc" Sep 18 06:36:35.284416 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:35.284283 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d43a82f2-a138-4b8e-917f-bf0b2ab4aa5c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hdknc\" (UID: \"d43a82f2-a138-4b8e-917f-bf0b2ab4aa5c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hdknc" Sep 18 06:36:35.284416 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:35.284350 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d43a82f2-a138-4b8e-917f-bf0b2ab4aa5c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hdknc\" (UID: \"d43a82f2-a138-4b8e-917f-bf0b2ab4aa5c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hdknc" Sep 18 06:36:35.385110 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:35.385076 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d43a82f2-a138-4b8e-917f-bf0b2ab4aa5c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hdknc\" (UID: \"d43a82f2-a138-4b8e-917f-bf0b2ab4aa5c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hdknc" Sep 18 06:36:35.385285 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:35.385140 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d43a82f2-a138-4b8e-917f-bf0b2ab4aa5c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hdknc\" (UID: \"d43a82f2-a138-4b8e-917f-bf0b2ab4aa5c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hdknc" Sep 18 06:36:35.385285 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:35.385170 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d43a82f2-a138-4b8e-917f-bf0b2ab4aa5c-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hdknc\" (UID: \"d43a82f2-a138-4b8e-917f-bf0b2ab4aa5c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hdknc" Sep 18 06:36:35.385285 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:35.385205 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvlxb\" (UniqueName: \"kubernetes.io/projected/d43a82f2-a138-4b8e-917f-bf0b2ab4aa5c-kube-api-access-pvlxb\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hdknc\" (UID: \"d43a82f2-a138-4b8e-917f-bf0b2ab4aa5c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hdknc" Sep 18 06:36:35.385285 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:35.385234 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d43a82f2-a138-4b8e-917f-bf0b2ab4aa5c-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hdknc\" (UID: \"d43a82f2-a138-4b8e-917f-bf0b2ab4aa5c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hdknc" Sep 18 06:36:35.385285 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:35.385266 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d43a82f2-a138-4b8e-917f-bf0b2ab4aa5c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hdknc\" (UID: \"d43a82f2-a138-4b8e-917f-bf0b2ab4aa5c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hdknc" Sep 18 06:36:35.385726 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:35.385702 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d43a82f2-a138-4b8e-917f-bf0b2ab4aa5c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hdknc\" (UID: \"d43a82f2-a138-4b8e-917f-bf0b2ab4aa5c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hdknc" Sep 18 06:36:35.385821 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:35.385732 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d43a82f2-a138-4b8e-917f-bf0b2ab4aa5c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hdknc\" (UID: \"d43a82f2-a138-4b8e-917f-bf0b2ab4aa5c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hdknc" Sep 18 06:36:35.385821 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:35.385788 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d43a82f2-a138-4b8e-917f-bf0b2ab4aa5c-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hdknc\" (UID: \"d43a82f2-a138-4b8e-917f-bf0b2ab4aa5c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hdknc" Sep 18 06:36:35.387712 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:35.387685 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d43a82f2-a138-4b8e-917f-bf0b2ab4aa5c-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hdknc\" (UID: \"d43a82f2-a138-4b8e-917f-bf0b2ab4aa5c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hdknc" Sep 18 06:36:35.387907 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:35.387889 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d43a82f2-a138-4b8e-917f-bf0b2ab4aa5c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hdknc\" (UID: \"d43a82f2-a138-4b8e-917f-bf0b2ab4aa5c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hdknc" Sep 18 06:36:35.393908 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:35.393887 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvlxb\" (UniqueName: \"kubernetes.io/projected/d43a82f2-a138-4b8e-917f-bf0b2ab4aa5c-kube-api-access-pvlxb\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-hdknc\" (UID: \"d43a82f2-a138-4b8e-917f-bf0b2ab4aa5c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hdknc" Sep 18 06:36:35.499199 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:35.499109 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hdknc" Sep 18 06:36:35.771179 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:35.771150 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hdknc"] Sep 18 06:36:35.773180 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:36:35.773157 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd43a82f2_a138_4b8e_917f_bf0b2ab4aa5c.slice/crio-695b8c45ed618391b8cf77d43aa948ba7ce028a4bcd5a18ead09c8937478b238 WatchSource:0}: Error finding container 695b8c45ed618391b8cf77d43aa948ba7ce028a4bcd5a18ead09c8937478b238: Status 404 returned error can't find the container with id 695b8c45ed618391b8cf77d43aa948ba7ce028a4bcd5a18ead09c8937478b238 Sep 18 06:36:36.058559 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:36.058469 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hdknc" event={"ID":"d43a82f2-a138-4b8e-917f-bf0b2ab4aa5c","Type":"ContainerStarted","Data":"695b8c45ed618391b8cf77d43aa948ba7ce028a4bcd5a18ead09c8937478b238"} Sep 18 06:36:37.067554 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:37.067522 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hdknc" event={"ID":"d43a82f2-a138-4b8e-917f-bf0b2ab4aa5c","Type":"ContainerStarted","Data":"217ffd2703a7ce483b6c8650683738e178ae8fc782bdc4d2b6edcca3929615ad"} Sep 18 06:36:37.085918 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:37.085868 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hdknc" podStartSLOduration=1.008021968 podStartE2EDuration="2.085850831s" totalImagesPullingTime="1.077828863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:36:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:36:37.083436813 +0000 UTC m=+858.822853014" watchObservedRunningTime="2026-09-18 06:36:37.085850831 +0000 UTC m=+858.825267005" Sep 18 06:36:45.499531 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:45.499496 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hdknc" Sep 18 06:36:45.500089 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:45.499550 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hdknc" Sep 18 06:36:45.511892 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:45.511869 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hdknc" Sep 18 06:36:46.126093 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:36:46.126067 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-hdknc" Sep 18 06:37:20.400716 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:37:20.400685 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjr4h_eb059951-e263-40da-9155-4fd0cf242820/ovn-acl-logging/0.log" Sep 18 06:37:20.402079 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:37:20.402061 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjr4h_eb059951-e263-40da-9155-4fd0cf242820/ovn-acl-logging/0.log" Sep 18 06:37:36.386592 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:37:36.386550 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-2j9qq"] Sep 18 06:37:36.389868 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:37:36.389853 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2j9qq" Sep 18 06:37:36.392040 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:37:36.392018 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Sep 18 06:37:36.399405 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:37:36.399379 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-2j9qq"] Sep 18 06:37:36.486576 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:37:36.486547 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d2b5190-5648-44e7-b595-8e6346a0343d-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2j9qq\" (UID: \"7d2b5190-5648-44e7-b595-8e6346a0343d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2j9qq" Sep 18 06:37:36.486748 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:37:36.486617 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d2b5190-5648-44e7-b595-8e6346a0343d-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2j9qq\" (UID: \"7d2b5190-5648-44e7-b595-8e6346a0343d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2j9qq" Sep 18 06:37:36.486748 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:37:36.486651 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d2b5190-5648-44e7-b595-8e6346a0343d-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2j9qq\" (UID: \"7d2b5190-5648-44e7-b595-8e6346a0343d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2j9qq" Sep 18 06:37:36.486748 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:37:36.486672 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl6hv\" (UniqueName: \"kubernetes.io/projected/7d2b5190-5648-44e7-b595-8e6346a0343d-kube-api-access-vl6hv\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2j9qq\" (UID: \"7d2b5190-5648-44e7-b595-8e6346a0343d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2j9qq" Sep 18 06:37:36.486748 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:37:36.486705 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d2b5190-5648-44e7-b595-8e6346a0343d-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2j9qq\" (UID: \"7d2b5190-5648-44e7-b595-8e6346a0343d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2j9qq" Sep 18 06:37:36.486893 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:37:36.486790 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d2b5190-5648-44e7-b595-8e6346a0343d-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2j9qq\" (UID: \"7d2b5190-5648-44e7-b595-8e6346a0343d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2j9qq" Sep 18 06:37:36.587811 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:37:36.587778 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d2b5190-5648-44e7-b595-8e6346a0343d-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2j9qq\" (UID: \"7d2b5190-5648-44e7-b595-8e6346a0343d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2j9qq" Sep 18 06:37:36.587984 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:37:36.587827 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d2b5190-5648-44e7-b595-8e6346a0343d-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2j9qq\" (UID: \"7d2b5190-5648-44e7-b595-8e6346a0343d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2j9qq" Sep 18 06:37:36.587984 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:37:36.587860 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d2b5190-5648-44e7-b595-8e6346a0343d-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2j9qq\" (UID: \"7d2b5190-5648-44e7-b595-8e6346a0343d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2j9qq" Sep 18 06:37:36.587984 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:37:36.587883 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl6hv\" (UniqueName: \"kubernetes.io/projected/7d2b5190-5648-44e7-b595-8e6346a0343d-kube-api-access-vl6hv\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2j9qq\" (UID: \"7d2b5190-5648-44e7-b595-8e6346a0343d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2j9qq" Sep 18 06:37:36.587984 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:37:36.587913 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d2b5190-5648-44e7-b595-8e6346a0343d-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2j9qq\" (UID: \"7d2b5190-5648-44e7-b595-8e6346a0343d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2j9qq" Sep 18 06:37:36.587984 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:37:36.587968 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d2b5190-5648-44e7-b595-8e6346a0343d-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2j9qq\" (UID: \"7d2b5190-5648-44e7-b595-8e6346a0343d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2j9qq" Sep 18 06:37:36.588261 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:37:36.588236 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d2b5190-5648-44e7-b595-8e6346a0343d-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2j9qq\" (UID: \"7d2b5190-5648-44e7-b595-8e6346a0343d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2j9qq" Sep 18 06:37:36.588324 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:37:36.588275 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d2b5190-5648-44e7-b595-8e6346a0343d-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2j9qq\" (UID: \"7d2b5190-5648-44e7-b595-8e6346a0343d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2j9qq" Sep 18 06:37:36.588459 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:37:36.588435 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d2b5190-5648-44e7-b595-8e6346a0343d-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2j9qq\" (UID: \"7d2b5190-5648-44e7-b595-8e6346a0343d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2j9qq" Sep 18 06:37:36.590108 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:37:36.590065 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d2b5190-5648-44e7-b595-8e6346a0343d-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2j9qq\" (UID: \"7d2b5190-5648-44e7-b595-8e6346a0343d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2j9qq" Sep 18 06:37:36.590476 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:37:36.590460 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d2b5190-5648-44e7-b595-8e6346a0343d-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2j9qq\" (UID: \"7d2b5190-5648-44e7-b595-8e6346a0343d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2j9qq" Sep 18 06:37:36.596048 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:37:36.596019 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl6hv\" (UniqueName: \"kubernetes.io/projected/7d2b5190-5648-44e7-b595-8e6346a0343d-kube-api-access-vl6hv\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2j9qq\" (UID: \"7d2b5190-5648-44e7-b595-8e6346a0343d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2j9qq" Sep 18 06:37:36.700182 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:37:36.700120 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2j9qq" Sep 18 06:37:36.968758 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:37:36.968725 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-2j9qq"] Sep 18 06:37:36.969887 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:37:36.969859 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d2b5190_5648_44e7_b595_8e6346a0343d.slice/crio-596955e614c7d1ef9ec3914ea4dbe43f44038b257dbc94ee156090d9a7cc83ad WatchSource:0}: Error finding container 596955e614c7d1ef9ec3914ea4dbe43f44038b257dbc94ee156090d9a7cc83ad: Status 404 returned error can't find the container with id 596955e614c7d1ef9ec3914ea4dbe43f44038b257dbc94ee156090d9a7cc83ad Sep 18 06:37:37.382324 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:37:37.382284 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2j9qq" event={"ID":"7d2b5190-5648-44e7-b595-8e6346a0343d","Type":"ContainerStarted","Data":"ce4288ec56bebc43b51f5cdf9249e80d3f4cc51d02e22ac4dad2b07885bf8f21"} Sep 18 06:37:37.382515 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:37:37.382340 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2j9qq" event={"ID":"7d2b5190-5648-44e7-b595-8e6346a0343d","Type":"ContainerStarted","Data":"596955e614c7d1ef9ec3914ea4dbe43f44038b257dbc94ee156090d9a7cc83ad"} Sep 18 06:37:37.399486 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:37:37.399439 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2j9qq" podStartSLOduration=1.22625439 podStartE2EDuration="1.399423712s" totalImagesPullingTime="173.169322ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:37:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:37:37.396908746 +0000 UTC m=+919.136324944" watchObservedRunningTime="2026-09-18 06:37:37.399423712 +0000 UTC m=+919.138839927" Sep 18 06:37:46.701232 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:37:46.701200 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2j9qq" Sep 18 06:37:46.701625 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:37:46.701281 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2j9qq" Sep 18 06:37:46.713553 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:37:46.713532 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2j9qq" Sep 18 06:37:47.453251 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:37:47.453225 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2j9qq" Sep 18 06:40:23.302038 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.301989 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262"] Sep 18 06:40:23.305621 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.305596 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262" Sep 18 06:40:23.308203 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.308184 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-654463-openshift-default-dockercfg-qkwrl\"" Sep 18 06:40:23.317454 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.317430 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262"] Sep 18 06:40:23.354060 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.354026 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8-workload-socket\") pod \"e2e-worker-w4-654463-openshift-default-8bf48d448-d6262\" (UID: \"5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8\") " pod="openshift-ingress/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262" Sep 18 06:40:23.354227 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.354128 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8-istio-data\") pod \"e2e-worker-w4-654463-openshift-default-8bf48d448-d6262\" (UID: \"5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8\") " pod="openshift-ingress/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262" Sep 18 06:40:23.354227 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.354194 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8-credential-socket\") pod \"e2e-worker-w4-654463-openshift-default-8bf48d448-d6262\" (UID: \"5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8\") " pod="openshift-ingress/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262" Sep 18 06:40:23.354396 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.354264 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8-workload-certs\") pod \"e2e-worker-w4-654463-openshift-default-8bf48d448-d6262\" (UID: \"5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8\") " pod="openshift-ingress/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262" Sep 18 06:40:23.354396 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.354290 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8-istio-podinfo\") pod \"e2e-worker-w4-654463-openshift-default-8bf48d448-d6262\" (UID: \"5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8\") " pod="openshift-ingress/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262" Sep 18 06:40:23.354396 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.354310 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8-istiod-ca-cert\") pod \"e2e-worker-w4-654463-openshift-default-8bf48d448-d6262\" (UID: \"5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8\") " pod="openshift-ingress/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262" Sep 18 06:40:23.354396 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.354343 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8-istio-token\") pod \"e2e-worker-w4-654463-openshift-default-8bf48d448-d6262\" (UID: \"5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8\") " pod="openshift-ingress/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262" Sep 18 06:40:23.354560 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.354478 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8-istio-envoy\") pod \"e2e-worker-w4-654463-openshift-default-8bf48d448-d6262\" (UID: \"5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8\") " pod="openshift-ingress/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262" Sep 18 06:40:23.354560 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.354518 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkzb6\" (UniqueName: \"kubernetes.io/projected/5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8-kube-api-access-wkzb6\") pod \"e2e-worker-w4-654463-openshift-default-8bf48d448-d6262\" (UID: \"5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8\") " pod="openshift-ingress/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262" Sep 18 06:40:23.367019 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.366991 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw"] Sep 18 06:40:23.371026 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.371005 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw" Sep 18 06:40:23.373800 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.373777 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-7c4f4b-openshift-default-dockercfg-2jgbv\"" Sep 18 06:40:23.383105 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.383080 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw"] Sep 18 06:40:23.455314 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.455285 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/504bef64-f0e4-4856-a251-9296702689c4-workload-socket\") pod \"e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw\" (UID: \"504bef64-f0e4-4856-a251-9296702689c4\") " pod="openshift-ingress/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw" Sep 18 06:40:23.455503 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.455335 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8-istio-data\") pod \"e2e-worker-w4-654463-openshift-default-8bf48d448-d6262\" (UID: \"5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8\") " pod="openshift-ingress/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262" Sep 18 06:40:23.455503 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.455424 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8-credential-socket\") pod \"e2e-worker-w4-654463-openshift-default-8bf48d448-d6262\" (UID: \"5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8\") " pod="openshift-ingress/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262" Sep 18 06:40:23.455503 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.455462 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/504bef64-f0e4-4856-a251-9296702689c4-istio-podinfo\") pod \"e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw\" (UID: \"504bef64-f0e4-4856-a251-9296702689c4\") " pod="openshift-ingress/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw" Sep 18 06:40:23.455658 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.455512 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/504bef64-f0e4-4856-a251-9296702689c4-credential-socket\") pod \"e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw\" (UID: \"504bef64-f0e4-4856-a251-9296702689c4\") " pod="openshift-ingress/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw" Sep 18 06:40:23.455658 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.455567 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8-workload-certs\") pod \"e2e-worker-w4-654463-openshift-default-8bf48d448-d6262\" (UID: \"5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8\") " pod="openshift-ingress/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262" Sep 18 06:40:23.455658 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.455592 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8-istio-podinfo\") pod \"e2e-worker-w4-654463-openshift-default-8bf48d448-d6262\" (UID: \"5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8\") " pod="openshift-ingress/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262" Sep 18 06:40:23.455658 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.455618 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8-istiod-ca-cert\") pod \"e2e-worker-w4-654463-openshift-default-8bf48d448-d6262\" (UID: \"5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8\") " pod="openshift-ingress/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262" Sep 18 06:40:23.455658 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.455652 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/504bef64-f0e4-4856-a251-9296702689c4-istio-envoy\") pod \"e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw\" (UID: \"504bef64-f0e4-4856-a251-9296702689c4\") " pod="openshift-ingress/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw" Sep 18 06:40:23.455921 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.455676 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8-istio-data\") pod \"e2e-worker-w4-654463-openshift-default-8bf48d448-d6262\" (UID: \"5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8\") " pod="openshift-ingress/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262" Sep 18 06:40:23.455921 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.455677 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/504bef64-f0e4-4856-a251-9296702689c4-istio-data\") pod \"e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw\" (UID: \"504bef64-f0e4-4856-a251-9296702689c4\") " pod="openshift-ingress/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw" Sep 18 06:40:23.455921 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.455732 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8-istio-token\") pod \"e2e-worker-w4-654463-openshift-default-8bf48d448-d6262\" (UID: \"5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8\") " pod="openshift-ingress/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262" Sep 18 06:40:23.455921 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.455750 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/504bef64-f0e4-4856-a251-9296702689c4-workload-certs\") pod \"e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw\" (UID: \"504bef64-f0e4-4856-a251-9296702689c4\") " pod="openshift-ingress/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw" Sep 18 06:40:23.455921 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.455761 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8-credential-socket\") pod \"e2e-worker-w4-654463-openshift-default-8bf48d448-d6262\" (UID: \"5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8\") " pod="openshift-ingress/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262" Sep 18 06:40:23.455921 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.455772 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/504bef64-f0e4-4856-a251-9296702689c4-istiod-ca-cert\") pod \"e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw\" (UID: \"504bef64-f0e4-4856-a251-9296702689c4\") " pod="openshift-ingress/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw" Sep 18 06:40:23.455921 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.455796 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw2r2\" (UniqueName: \"kubernetes.io/projected/504bef64-f0e4-4856-a251-9296702689c4-kube-api-access-bw2r2\") pod \"e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw\" (UID: \"504bef64-f0e4-4856-a251-9296702689c4\") " pod="openshift-ingress/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw" Sep 18 06:40:23.455921 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.455912 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8-istio-envoy\") pod \"e2e-worker-w4-654463-openshift-default-8bf48d448-d6262\" (UID: \"5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8\") " pod="openshift-ingress/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262" Sep 18 06:40:23.456267 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.455935 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8-workload-certs\") pod \"e2e-worker-w4-654463-openshift-default-8bf48d448-d6262\" (UID: \"5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8\") " pod="openshift-ingress/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262" Sep 18 06:40:23.456267 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.455952 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkzb6\" (UniqueName: \"kubernetes.io/projected/5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8-kube-api-access-wkzb6\") pod \"e2e-worker-w4-654463-openshift-default-8bf48d448-d6262\" (UID: \"5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8\") " pod="openshift-ingress/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262" Sep 18 06:40:23.456267 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.456031 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/504bef64-f0e4-4856-a251-9296702689c4-istio-token\") pod \"e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw\" (UID: \"504bef64-f0e4-4856-a251-9296702689c4\") " pod="openshift-ingress/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw" Sep 18 06:40:23.456267 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.456084 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8-workload-socket\") pod \"e2e-worker-w4-654463-openshift-default-8bf48d448-d6262\" (UID: \"5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8\") " pod="openshift-ingress/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262" Sep 18 06:40:23.456447 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.456427 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8-workload-socket\") pod \"e2e-worker-w4-654463-openshift-default-8bf48d448-d6262\" (UID: \"5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8\") " pod="openshift-ingress/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262" Sep 18 06:40:23.456833 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.456812 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8-istiod-ca-cert\") pod \"e2e-worker-w4-654463-openshift-default-8bf48d448-d6262\" (UID: \"5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8\") " pod="openshift-ingress/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262" Sep 18 06:40:23.458113 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.458089 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8-istio-envoy\") pod \"e2e-worker-w4-654463-openshift-default-8bf48d448-d6262\" (UID: \"5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8\") " pod="openshift-ingress/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262" Sep 18 06:40:23.458418 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.458357 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8-istio-podinfo\") pod \"e2e-worker-w4-654463-openshift-default-8bf48d448-d6262\" (UID: \"5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8\") " pod="openshift-ingress/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262" Sep 18 06:40:23.465819 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.465786 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8-istio-token\") pod \"e2e-worker-w4-654463-openshift-default-8bf48d448-d6262\" (UID: \"5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8\") " pod="openshift-ingress/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262" Sep 18 06:40:23.465985 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.465966 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkzb6\" (UniqueName: \"kubernetes.io/projected/5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8-kube-api-access-wkzb6\") pod \"e2e-worker-w4-654463-openshift-default-8bf48d448-d6262\" (UID: \"5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8\") " pod="openshift-ingress/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262" Sep 18 06:40:23.557129 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.557053 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/504bef64-f0e4-4856-a251-9296702689c4-istio-envoy\") pod \"e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw\" (UID: \"504bef64-f0e4-4856-a251-9296702689c4\") " pod="openshift-ingress/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw" Sep 18 06:40:23.557129 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.557087 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/504bef64-f0e4-4856-a251-9296702689c4-istio-data\") pod \"e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw\" (UID: \"504bef64-f0e4-4856-a251-9296702689c4\") " pod="openshift-ingress/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw" Sep 18 06:40:23.557129 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.557119 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/504bef64-f0e4-4856-a251-9296702689c4-workload-certs\") pod \"e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw\" (UID: \"504bef64-f0e4-4856-a251-9296702689c4\") " pod="openshift-ingress/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw" Sep 18 06:40:23.557420 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.557144 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/504bef64-f0e4-4856-a251-9296702689c4-istiod-ca-cert\") pod \"e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw\" (UID: \"504bef64-f0e4-4856-a251-9296702689c4\") " pod="openshift-ingress/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw" Sep 18 06:40:23.557420 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.557167 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw2r2\" (UniqueName: \"kubernetes.io/projected/504bef64-f0e4-4856-a251-9296702689c4-kube-api-access-bw2r2\") pod \"e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw\" (UID: \"504bef64-f0e4-4856-a251-9296702689c4\") " pod="openshift-ingress/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw" Sep 18 06:40:23.558173 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.558149 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/504bef64-f0e4-4856-a251-9296702689c4-istio-token\") pod \"e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw\" (UID: \"504bef64-f0e4-4856-a251-9296702689c4\") " pod="openshift-ingress/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw" Sep 18 06:40:23.558389 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.558343 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/504bef64-f0e4-4856-a251-9296702689c4-workload-socket\") pod \"e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw\" (UID: \"504bef64-f0e4-4856-a251-9296702689c4\") " pod="openshift-ingress/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw" Sep 18 06:40:23.558512 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.558268 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/504bef64-f0e4-4856-a251-9296702689c4-istio-data\") pod \"e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw\" (UID: \"504bef64-f0e4-4856-a251-9296702689c4\") " pod="openshift-ingress/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw" Sep 18 06:40:23.558512 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.558401 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/504bef64-f0e4-4856-a251-9296702689c4-istiod-ca-cert\") pod \"e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw\" (UID: \"504bef64-f0e4-4856-a251-9296702689c4\") " pod="openshift-ingress/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw" Sep 18 06:40:23.558512 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.558452 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/504bef64-f0e4-4856-a251-9296702689c4-istio-podinfo\") pod \"e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw\" (UID: \"504bef64-f0e4-4856-a251-9296702689c4\") " pod="openshift-ingress/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw" Sep 18 06:40:23.558512 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.558499 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/504bef64-f0e4-4856-a251-9296702689c4-credential-socket\") pod \"e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw\" (UID: \"504bef64-f0e4-4856-a251-9296702689c4\") " pod="openshift-ingress/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw" Sep 18 06:40:23.558949 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.558923 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/504bef64-f0e4-4856-a251-9296702689c4-workload-certs\") pod \"e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw\" (UID: \"504bef64-f0e4-4856-a251-9296702689c4\") " pod="openshift-ingress/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw" Sep 18 06:40:23.559040 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.558944 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/504bef64-f0e4-4856-a251-9296702689c4-workload-socket\") pod \"e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw\" (UID: \"504bef64-f0e4-4856-a251-9296702689c4\") " pod="openshift-ingress/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw" Sep 18 06:40:23.559040 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.558987 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/504bef64-f0e4-4856-a251-9296702689c4-credential-socket\") pod \"e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw\" (UID: \"504bef64-f0e4-4856-a251-9296702689c4\") " pod="openshift-ingress/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw" Sep 18 06:40:23.559801 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.559775 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/504bef64-f0e4-4856-a251-9296702689c4-istio-envoy\") pod \"e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw\" (UID: \"504bef64-f0e4-4856-a251-9296702689c4\") " pod="openshift-ingress/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw" Sep 18 06:40:23.561023 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.561006 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/504bef64-f0e4-4856-a251-9296702689c4-istio-podinfo\") pod \"e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw\" (UID: \"504bef64-f0e4-4856-a251-9296702689c4\") " pod="openshift-ingress/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw" Sep 18 06:40:23.565205 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.565181 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/504bef64-f0e4-4856-a251-9296702689c4-istio-token\") pod \"e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw\" (UID: \"504bef64-f0e4-4856-a251-9296702689c4\") " pod="openshift-ingress/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw" Sep 18 06:40:23.565503 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.565464 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw2r2\" (UniqueName: \"kubernetes.io/projected/504bef64-f0e4-4856-a251-9296702689c4-kube-api-access-bw2r2\") pod \"e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw\" (UID: \"504bef64-f0e4-4856-a251-9296702689c4\") " pod="openshift-ingress/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw" Sep 18 06:40:23.616867 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.616842 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262" Sep 18 06:40:23.685843 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.685814 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw" Sep 18 06:40:23.899529 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.899502 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262"] Sep 18 06:40:23.900584 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:40:23.900556 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a8b52a3_9d9a_4a7c_bfb7_5faca89c2cc8.slice/crio-d8ca0bbb2c2beebc33210d6d4291c3caca32708e604c1de2264524fde9739fc9 WatchSource:0}: Error finding container d8ca0bbb2c2beebc33210d6d4291c3caca32708e604c1de2264524fde9739fc9: Status 404 returned error can't find the container with id d8ca0bbb2c2beebc33210d6d4291c3caca32708e604c1de2264524fde9739fc9 Sep 18 06:40:23.902794 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.902763 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:40:23.902877 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.902827 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:40:23.902877 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.902853 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:40:23.964003 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.963980 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw"] Sep 18 06:40:23.964581 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:40:23.964559 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod504bef64_f0e4_4856_a251_9296702689c4.slice/crio-214d3c91b99a6d8993a819e8586a8a81e7df7881248cdde89c899ac09810f596 WatchSource:0}: Error finding container 214d3c91b99a6d8993a819e8586a8a81e7df7881248cdde89c899ac09810f596: Status 404 returned error can't find the container with id 214d3c91b99a6d8993a819e8586a8a81e7df7881248cdde89c899ac09810f596 Sep 18 06:40:23.966700 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.966666 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:40:23.966766 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.966751 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:40:23.966817 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:23.966799 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:40:24.274666 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:24.274587 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262" event={"ID":"5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8","Type":"ContainerStarted","Data":"d45ebdf186426c3e3c1691771b762497d85f1643093e5d0f4fc83446874b6dca"} Sep 18 06:40:24.274666 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:24.274640 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262" event={"ID":"5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8","Type":"ContainerStarted","Data":"d8ca0bbb2c2beebc33210d6d4291c3caca32708e604c1de2264524fde9739fc9"} Sep 18 06:40:24.276220 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:24.276190 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw" event={"ID":"504bef64-f0e4-4856-a251-9296702689c4","Type":"ContainerStarted","Data":"280d89efcbcbad5984a021c0fac52bcf28d945b59cb8c97b0494623e0cdc1835"} Sep 18 06:40:24.276345 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:24.276246 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw" event={"ID":"504bef64-f0e4-4856-a251-9296702689c4","Type":"ContainerStarted","Data":"214d3c91b99a6d8993a819e8586a8a81e7df7881248cdde89c899ac09810f596"} Sep 18 06:40:24.294309 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:24.294267 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262" podStartSLOduration=1.2942516020000001 podStartE2EDuration="1.294251602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:40:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:40:24.294210311 +0000 UTC m=+1086.033626507" watchObservedRunningTime="2026-09-18 06:40:24.294251602 +0000 UTC m=+1086.033667776" Sep 18 06:40:24.316144 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:24.316097 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw" podStartSLOduration=1.316082714 podStartE2EDuration="1.316082714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:40:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:40:24.313144763 +0000 UTC m=+1086.052560974" watchObservedRunningTime="2026-09-18 06:40:24.316082714 +0000 UTC m=+1086.055498888" Sep 18 06:40:24.617764 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:24.617731 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262" Sep 18 06:40:24.622660 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:24.622636 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262" Sep 18 06:40:24.686463 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:24.686442 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw" Sep 18 06:40:25.281543 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:25.281508 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262" Sep 18 06:40:25.282236 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:25.282218 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262" Sep 18 06:40:25.690743 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:25.690715 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw" Sep 18 06:40:26.287659 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:26.287626 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw" Sep 18 06:40:26.288538 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:26.288522 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw" Sep 18 06:40:28.684570 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:28.684536 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-654463-db8fb8547-x6klj"] Sep 18 06:40:28.688630 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:28.688608 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-654463-db8fb8547-x6klj" Sep 18 06:40:28.691176 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:28.691153 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-654463\"" Sep 18 06:40:28.691281 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:28.691215 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-654463-dockercfg-jddxm\"" Sep 18 06:40:28.701202 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:28.701179 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-654463-db8fb8547-x6klj"] Sep 18 06:40:28.811096 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:28.811065 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5wrq\" (UniqueName: \"kubernetes.io/projected/885403b9-b7b0-4cc0-aa50-9486ba22a99d-kube-api-access-v5wrq\") pod \"payload-processing-e2e-worker-w4-654463-db8fb8547-x6klj\" (UID: \"885403b9-b7b0-4cc0-aa50-9486ba22a99d\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-654463-db8fb8547-x6klj" Sep 18 06:40:28.811263 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:28.811117 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/885403b9-b7b0-4cc0-aa50-9486ba22a99d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-654463-db8fb8547-x6klj\" (UID: \"885403b9-b7b0-4cc0-aa50-9486ba22a99d\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-654463-db8fb8547-x6klj" Sep 18 06:40:28.912522 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:28.912491 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5wrq\" (UniqueName: \"kubernetes.io/projected/885403b9-b7b0-4cc0-aa50-9486ba22a99d-kube-api-access-v5wrq\") pod \"payload-processing-e2e-worker-w4-654463-db8fb8547-x6klj\" (UID: \"885403b9-b7b0-4cc0-aa50-9486ba22a99d\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-654463-db8fb8547-x6klj" Sep 18 06:40:28.912696 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:28.912539 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/885403b9-b7b0-4cc0-aa50-9486ba22a99d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-654463-db8fb8547-x6klj\" (UID: \"885403b9-b7b0-4cc0-aa50-9486ba22a99d\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-654463-db8fb8547-x6klj" Sep 18 06:40:28.913261 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:28.913240 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/885403b9-b7b0-4cc0-aa50-9486ba22a99d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-654463-db8fb8547-x6klj\" (UID: \"885403b9-b7b0-4cc0-aa50-9486ba22a99d\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-654463-db8fb8547-x6klj" Sep 18 06:40:28.927741 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:28.927706 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5wrq\" (UniqueName: \"kubernetes.io/projected/885403b9-b7b0-4cc0-aa50-9486ba22a99d-kube-api-access-v5wrq\") pod \"payload-processing-e2e-worker-w4-654463-db8fb8547-x6klj\" (UID: \"885403b9-b7b0-4cc0-aa50-9486ba22a99d\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-654463-db8fb8547-x6klj" Sep 18 06:40:29.001425 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:29.001329 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-654463-db8fb8547-x6klj" Sep 18 06:40:29.277885 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:29.277860 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-654463-db8fb8547-x6klj"] Sep 18 06:40:29.283187 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:40:29.280900 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod885403b9_b7b0_4cc0_aa50_9486ba22a99d.slice/crio-2a8667755c64de5d27eb324c9afd42731c547025b824b24fb2d19979ab0a2e3c WatchSource:0}: Error finding container 2a8667755c64de5d27eb324c9afd42731c547025b824b24fb2d19979ab0a2e3c: Status 404 returned error can't find the container with id 2a8667755c64de5d27eb324c9afd42731c547025b824b24fb2d19979ab0a2e3c Sep 18 06:40:29.306430 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:29.306398 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-654463-db8fb8547-x6klj" event={"ID":"885403b9-b7b0-4cc0-aa50-9486ba22a99d","Type":"ContainerStarted","Data":"2a8667755c64de5d27eb324c9afd42731c547025b824b24fb2d19979ab0a2e3c"} Sep 18 06:40:30.314640 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:30.314602 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-654463-db8fb8547-x6klj" event={"ID":"885403b9-b7b0-4cc0-aa50-9486ba22a99d","Type":"ContainerStarted","Data":"79ee1af26893305254fac413f967d0b1c14740c99c82ae74994cd1e148e265b3"} Sep 18 06:40:30.315036 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:30.314665 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-654463-db8fb8547-x6klj" Sep 18 06:40:30.332848 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:30.332791 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-654463-db8fb8547-x6klj" podStartSLOduration=2.332774286 podStartE2EDuration="2.332774286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:40:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:40:30.332209705 +0000 UTC m=+1092.071625923" watchObservedRunningTime="2026-09-18 06:40:30.332774286 +0000 UTC m=+1092.072190481" Sep 18 06:40:30.595812 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:30.595777 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-946609-67f4478978-fmbp7"] Sep 18 06:40:30.599484 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:30.599465 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-946609-67f4478978-fmbp7" Sep 18 06:40:30.602446 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:30.602425 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-946609-dockercfg-xgzbz\"" Sep 18 06:40:30.602728 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:30.602465 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-946609\"" Sep 18 06:40:30.606356 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:30.606335 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-946609-67f4478978-fmbp7"] Sep 18 06:40:30.628147 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:30.628125 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-946609-9fd88897f-68ljl"] Sep 18 06:40:30.631602 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:30.631581 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-946609-9fd88897f-68ljl" Sep 18 06:40:30.639356 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:30.639334 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-946609-9fd88897f-68ljl"] Sep 18 06:40:30.729821 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:30.729785 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcc7p\" (UniqueName: \"kubernetes.io/projected/feb1276d-6f0b-4bb0-99f6-d64a45b0a8d0-kube-api-access-vcc7p\") pod \"payload-pre-processing-e2e-worker-w0-946609-67f4478978-fmbp7\" (UID: \"feb1276d-6f0b-4bb0-99f6-d64a45b0a8d0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-946609-67f4478978-fmbp7" Sep 18 06:40:30.730007 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:30.729832 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14c2ea04-c360-4273-8aff-ebaadd21df0c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-946609-9fd88897f-68ljl\" (UID: \"14c2ea04-c360-4273-8aff-ebaadd21df0c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-946609-9fd88897f-68ljl" Sep 18 06:40:30.730007 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:30.729934 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ccc2\" (UniqueName: \"kubernetes.io/projected/14c2ea04-c360-4273-8aff-ebaadd21df0c-kube-api-access-4ccc2\") pod \"payload-processing-e2e-worker-w0-946609-9fd88897f-68ljl\" (UID: \"14c2ea04-c360-4273-8aff-ebaadd21df0c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-946609-9fd88897f-68ljl" Sep 18 06:40:30.730007 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:30.729952 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feb1276d-6f0b-4bb0-99f6-d64a45b0a8d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-946609-67f4478978-fmbp7\" (UID: \"feb1276d-6f0b-4bb0-99f6-d64a45b0a8d0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-946609-67f4478978-fmbp7" Sep 18 06:40:30.831397 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:30.830839 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ccc2\" (UniqueName: \"kubernetes.io/projected/14c2ea04-c360-4273-8aff-ebaadd21df0c-kube-api-access-4ccc2\") pod \"payload-processing-e2e-worker-w0-946609-9fd88897f-68ljl\" (UID: \"14c2ea04-c360-4273-8aff-ebaadd21df0c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-946609-9fd88897f-68ljl" Sep 18 06:40:30.831397 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:30.830895 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feb1276d-6f0b-4bb0-99f6-d64a45b0a8d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-946609-67f4478978-fmbp7\" (UID: \"feb1276d-6f0b-4bb0-99f6-d64a45b0a8d0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-946609-67f4478978-fmbp7" Sep 18 06:40:30.831397 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:30.830969 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcc7p\" (UniqueName: \"kubernetes.io/projected/feb1276d-6f0b-4bb0-99f6-d64a45b0a8d0-kube-api-access-vcc7p\") pod \"payload-pre-processing-e2e-worker-w0-946609-67f4478978-fmbp7\" (UID: \"feb1276d-6f0b-4bb0-99f6-d64a45b0a8d0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-946609-67f4478978-fmbp7" Sep 18 06:40:30.831397 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:30.831010 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14c2ea04-c360-4273-8aff-ebaadd21df0c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-946609-9fd88897f-68ljl\" (UID: \"14c2ea04-c360-4273-8aff-ebaadd21df0c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-946609-9fd88897f-68ljl" Sep 18 06:40:30.831896 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:30.831867 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14c2ea04-c360-4273-8aff-ebaadd21df0c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-946609-9fd88897f-68ljl\" (UID: \"14c2ea04-c360-4273-8aff-ebaadd21df0c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-946609-9fd88897f-68ljl" Sep 18 06:40:30.832011 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:30.831867 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feb1276d-6f0b-4bb0-99f6-d64a45b0a8d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-946609-67f4478978-fmbp7\" (UID: \"feb1276d-6f0b-4bb0-99f6-d64a45b0a8d0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-946609-67f4478978-fmbp7" Sep 18 06:40:30.841879 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:30.841852 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcc7p\" (UniqueName: \"kubernetes.io/projected/feb1276d-6f0b-4bb0-99f6-d64a45b0a8d0-kube-api-access-vcc7p\") pod \"payload-pre-processing-e2e-worker-w0-946609-67f4478978-fmbp7\" (UID: \"feb1276d-6f0b-4bb0-99f6-d64a45b0a8d0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-946609-67f4478978-fmbp7" Sep 18 06:40:30.842189 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:30.842172 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ccc2\" (UniqueName: \"kubernetes.io/projected/14c2ea04-c360-4273-8aff-ebaadd21df0c-kube-api-access-4ccc2\") pod \"payload-processing-e2e-worker-w0-946609-9fd88897f-68ljl\" (UID: \"14c2ea04-c360-4273-8aff-ebaadd21df0c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-946609-9fd88897f-68ljl" Sep 18 06:40:30.911861 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:30.911800 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-946609-67f4478978-fmbp7" Sep 18 06:40:30.943147 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:30.943114 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-946609-9fd88897f-68ljl" Sep 18 06:40:31.189276 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:31.189249 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-946609-67f4478978-fmbp7"] Sep 18 06:40:31.190560 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:40:31.190538 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeb1276d_6f0b_4bb0_99f6_d64a45b0a8d0.slice/crio-6ccf8f33e497812ba3243b0c57b46a7e5e7c44370e70a1341c010fc78b977f80 WatchSource:0}: Error finding container 6ccf8f33e497812ba3243b0c57b46a7e5e7c44370e70a1341c010fc78b977f80: Status 404 returned error can't find the container with id 6ccf8f33e497812ba3243b0c57b46a7e5e7c44370e70a1341c010fc78b977f80 Sep 18 06:40:31.228753 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:31.228726 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-946609-9fd88897f-68ljl"] Sep 18 06:40:31.230036 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:40:31.230013 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14c2ea04_c360_4273_8aff_ebaadd21df0c.slice/crio-0c13f49c7a463edeabd3660619527c8983f3e313050bee4cc3c735fed3f4358b WatchSource:0}: Error finding container 0c13f49c7a463edeabd3660619527c8983f3e313050bee4cc3c735fed3f4358b: Status 404 returned error can't find the container with id 0c13f49c7a463edeabd3660619527c8983f3e313050bee4cc3c735fed3f4358b Sep 18 06:40:31.323174 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:31.323146 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-946609-9fd88897f-68ljl" event={"ID":"14c2ea04-c360-4273-8aff-ebaadd21df0c","Type":"ContainerStarted","Data":"abb124f0e6e423a13fe2431aed9a450b37670839e966c06494d9396835aaee4d"} Sep 18 06:40:31.323584 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:31.323206 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-946609-9fd88897f-68ljl" event={"ID":"14c2ea04-c360-4273-8aff-ebaadd21df0c","Type":"ContainerStarted","Data":"0c13f49c7a463edeabd3660619527c8983f3e313050bee4cc3c735fed3f4358b"} Sep 18 06:40:31.323584 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:31.323239 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-946609-9fd88897f-68ljl" Sep 18 06:40:31.325132 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:31.325100 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-946609-67f4478978-fmbp7" event={"ID":"feb1276d-6f0b-4bb0-99f6-d64a45b0a8d0","Type":"ContainerStarted","Data":"e347cf8e4b5645d6bb5127848408b1dc5f06e6abd066066f031cb94cc70a9d0e"} Sep 18 06:40:31.325217 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:31.325148 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-946609-67f4478978-fmbp7" event={"ID":"feb1276d-6f0b-4bb0-99f6-d64a45b0a8d0","Type":"ContainerStarted","Data":"6ccf8f33e497812ba3243b0c57b46a7e5e7c44370e70a1341c010fc78b977f80"} Sep 18 06:40:31.325217 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:31.325177 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-946609-67f4478978-fmbp7" Sep 18 06:40:31.338279 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:31.338236 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-946609-9fd88897f-68ljl" podStartSLOduration=1.338221154 podStartE2EDuration="1.338221154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:40:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:40:31.337210049 +0000 UTC m=+1093.076626237" watchObservedRunningTime="2026-09-18 06:40:31.338221154 +0000 UTC m=+1093.077637328" Sep 18 06:40:31.355514 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:31.355456 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-946609-67f4478978-fmbp7" podStartSLOduration=1.355435827 podStartE2EDuration="1.355435827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:40:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:40:31.351925005 +0000 UTC m=+1093.091341205" watchObservedRunningTime="2026-09-18 06:40:31.355435827 +0000 UTC m=+1093.094852004" Sep 18 06:40:38.984390 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:38.983893 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0da33e9c-6974b99bfd-cvqm8"] Sep 18 06:40:38.993622 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:38.991563 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0da33e9c-6974b99bfd-cvqm8" Sep 18 06:40:38.994923 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:38.994884 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-0da33e9c\"" Sep 18 06:40:38.996130 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:38.995135 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-0da33e9c-dockercfg-b8x2h\"" Sep 18 06:40:39.006093 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:39.005416 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0da33e9c-6974b99bfd-cvqm8"] Sep 18 06:40:39.023445 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:39.023415 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0da33e9c-68795bc8cf-qqsg9"] Sep 18 06:40:39.027002 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:39.026977 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0da33e9c-68795bc8cf-qqsg9" Sep 18 06:40:39.034387 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:39.034351 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0da33e9c-68795bc8cf-qqsg9"] Sep 18 06:40:39.117282 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:39.117236 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8kfz\" (UniqueName: \"kubernetes.io/projected/ad0ef75d-4d0f-4295-9890-b14f9e55dc54-kube-api-access-x8kfz\") pod \"payload-processing-e2e-ait-0da33e9c-68795bc8cf-qqsg9\" (UID: \"ad0ef75d-4d0f-4295-9890-b14f9e55dc54\") " pod="openshift-ingress/payload-processing-e2e-ait-0da33e9c-68795bc8cf-qqsg9" Sep 18 06:40:39.117282 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:39.117281 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad0ef75d-4d0f-4295-9890-b14f9e55dc54-plugins-config-volume\") pod \"payload-processing-e2e-ait-0da33e9c-68795bc8cf-qqsg9\" (UID: \"ad0ef75d-4d0f-4295-9890-b14f9e55dc54\") " pod="openshift-ingress/payload-processing-e2e-ait-0da33e9c-68795bc8cf-qqsg9" Sep 18 06:40:39.117556 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:39.117317 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2f576ba-7293-4e5f-8cbf-e346ddf82310-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0da33e9c-6974b99bfd-cvqm8\" (UID: \"a2f576ba-7293-4e5f-8cbf-e346ddf82310\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0da33e9c-6974b99bfd-cvqm8" Sep 18 06:40:39.117556 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:39.117436 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwrbd\" (UniqueName: \"kubernetes.io/projected/a2f576ba-7293-4e5f-8cbf-e346ddf82310-kube-api-access-xwrbd\") pod \"payload-pre-processing-e2e-ait-0da33e9c-6974b99bfd-cvqm8\" (UID: \"a2f576ba-7293-4e5f-8cbf-e346ddf82310\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0da33e9c-6974b99bfd-cvqm8" Sep 18 06:40:39.218673 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:39.218635 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwrbd\" (UniqueName: \"kubernetes.io/projected/a2f576ba-7293-4e5f-8cbf-e346ddf82310-kube-api-access-xwrbd\") pod \"payload-pre-processing-e2e-ait-0da33e9c-6974b99bfd-cvqm8\" (UID: \"a2f576ba-7293-4e5f-8cbf-e346ddf82310\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0da33e9c-6974b99bfd-cvqm8" Sep 18 06:40:39.218881 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:39.218698 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8kfz\" (UniqueName: \"kubernetes.io/projected/ad0ef75d-4d0f-4295-9890-b14f9e55dc54-kube-api-access-x8kfz\") pod \"payload-processing-e2e-ait-0da33e9c-68795bc8cf-qqsg9\" (UID: \"ad0ef75d-4d0f-4295-9890-b14f9e55dc54\") " pod="openshift-ingress/payload-processing-e2e-ait-0da33e9c-68795bc8cf-qqsg9" Sep 18 06:40:39.218881 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:39.218728 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad0ef75d-4d0f-4295-9890-b14f9e55dc54-plugins-config-volume\") pod \"payload-processing-e2e-ait-0da33e9c-68795bc8cf-qqsg9\" (UID: \"ad0ef75d-4d0f-4295-9890-b14f9e55dc54\") " pod="openshift-ingress/payload-processing-e2e-ait-0da33e9c-68795bc8cf-qqsg9" Sep 18 06:40:39.218881 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:39.218794 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2f576ba-7293-4e5f-8cbf-e346ddf82310-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0da33e9c-6974b99bfd-cvqm8\" (UID: \"a2f576ba-7293-4e5f-8cbf-e346ddf82310\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0da33e9c-6974b99bfd-cvqm8" Sep 18 06:40:39.219572 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:39.219551 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2f576ba-7293-4e5f-8cbf-e346ddf82310-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0da33e9c-6974b99bfd-cvqm8\" (UID: \"a2f576ba-7293-4e5f-8cbf-e346ddf82310\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0da33e9c-6974b99bfd-cvqm8" Sep 18 06:40:39.219625 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:39.219576 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad0ef75d-4d0f-4295-9890-b14f9e55dc54-plugins-config-volume\") pod \"payload-processing-e2e-ait-0da33e9c-68795bc8cf-qqsg9\" (UID: \"ad0ef75d-4d0f-4295-9890-b14f9e55dc54\") " pod="openshift-ingress/payload-processing-e2e-ait-0da33e9c-68795bc8cf-qqsg9" Sep 18 06:40:39.227342 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:39.227316 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwrbd\" (UniqueName: \"kubernetes.io/projected/a2f576ba-7293-4e5f-8cbf-e346ddf82310-kube-api-access-xwrbd\") pod \"payload-pre-processing-e2e-ait-0da33e9c-6974b99bfd-cvqm8\" (UID: \"a2f576ba-7293-4e5f-8cbf-e346ddf82310\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0da33e9c-6974b99bfd-cvqm8" Sep 18 06:40:39.227507 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:39.227407 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8kfz\" (UniqueName: \"kubernetes.io/projected/ad0ef75d-4d0f-4295-9890-b14f9e55dc54-kube-api-access-x8kfz\") pod \"payload-processing-e2e-ait-0da33e9c-68795bc8cf-qqsg9\" (UID: \"ad0ef75d-4d0f-4295-9890-b14f9e55dc54\") " pod="openshift-ingress/payload-processing-e2e-ait-0da33e9c-68795bc8cf-qqsg9" Sep 18 06:40:39.311125 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:39.311046 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0da33e9c-6974b99bfd-cvqm8" Sep 18 06:40:39.338873 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:39.338846 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0da33e9c-68795bc8cf-qqsg9" Sep 18 06:40:39.639411 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:39.639356 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0da33e9c-6974b99bfd-cvqm8"] Sep 18 06:40:39.640857 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:40:39.640834 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2f576ba_7293_4e5f_8cbf_e346ddf82310.slice/crio-381a5ebb332baa9f0fef38f6f5e958c8085292a3afc6219996c0755cd6dbb3bc WatchSource:0}: Error finding container 381a5ebb332baa9f0fef38f6f5e958c8085292a3afc6219996c0755cd6dbb3bc: Status 404 returned error can't find the container with id 381a5ebb332baa9f0fef38f6f5e958c8085292a3afc6219996c0755cd6dbb3bc Sep 18 06:40:39.683745 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:39.677705 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0da33e9c-68795bc8cf-qqsg9"] Sep 18 06:40:40.420213 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:40.419294 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0da33e9c-68795bc8cf-qqsg9" event={"ID":"ad0ef75d-4d0f-4295-9890-b14f9e55dc54","Type":"ContainerStarted","Data":"0aecd3947dda63b7b3ea961fe7393d7a46cc192d54edfeffdf837b1086fc2a1c"} Sep 18 06:40:40.420213 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:40.419769 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0da33e9c-68795bc8cf-qqsg9" event={"ID":"ad0ef75d-4d0f-4295-9890-b14f9e55dc54","Type":"ContainerStarted","Data":"e8742c3db38cb11dcb049024d2055ba68fc9ee699937abf7da03fb2075986890"} Sep 18 06:40:40.420213 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:40.419825 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-0da33e9c-68795bc8cf-qqsg9" Sep 18 06:40:40.421717 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:40.421692 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0da33e9c-6974b99bfd-cvqm8" event={"ID":"a2f576ba-7293-4e5f-8cbf-e346ddf82310","Type":"ContainerStarted","Data":"1e34ea23fa97026c477c21290acd875e51a3b6db60a454d11d1b135fafc6c6ca"} Sep 18 06:40:40.421889 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:40.421872 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0da33e9c-6974b99bfd-cvqm8" event={"ID":"a2f576ba-7293-4e5f-8cbf-e346ddf82310","Type":"ContainerStarted","Data":"381a5ebb332baa9f0fef38f6f5e958c8085292a3afc6219996c0755cd6dbb3bc"} Sep 18 06:40:40.422631 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:40.422606 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0da33e9c-6974b99bfd-cvqm8" Sep 18 06:40:40.436711 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:40.436673 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-0da33e9c-68795bc8cf-qqsg9" podStartSLOduration=2.436661315 podStartE2EDuration="2.436661315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:40:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:40:40.435581034 +0000 UTC m=+1102.174997225" watchObservedRunningTime="2026-09-18 06:40:40.436661315 +0000 UTC m=+1102.176077488" Sep 18 06:40:40.472390 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:40.471328 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-0da33e9c-6974b99bfd-cvqm8" podStartSLOduration=2.471306572 podStartE2EDuration="2.471306572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:40:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:40:40.45568386 +0000 UTC m=+1102.195100051" watchObservedRunningTime="2026-09-18 06:40:40.471306572 +0000 UTC m=+1102.210722746" Sep 18 06:40:40.720467 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:40.720191 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-43fdfe-6bbcb85f7-wv7k8"] Sep 18 06:40:40.724884 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:40.724857 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-43fdfe-6bbcb85f7-wv7k8" Sep 18 06:40:40.728821 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:40.726881 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-43fdfe-6bbcb85f7-wv7k8"] Sep 18 06:40:40.728821 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:40.727656 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-43fdfe-dockercfg-b8lhc\"" Sep 18 06:40:40.728821 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:40.728223 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-43fdfe\"" Sep 18 06:40:40.751834 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:40.750849 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80a7d765-9b80-49f8-8d64-f72ec2acb5d7-plugins-config-volume\") pod \"payload-processing-e2e-autores-43fdfe-6bbcb85f7-wv7k8\" (UID: \"80a7d765-9b80-49f8-8d64-f72ec2acb5d7\") " pod="openshift-ingress/payload-processing-e2e-autores-43fdfe-6bbcb85f7-wv7k8" Sep 18 06:40:40.751834 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:40.750942 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knk25\" (UniqueName: \"kubernetes.io/projected/80a7d765-9b80-49f8-8d64-f72ec2acb5d7-kube-api-access-knk25\") pod \"payload-processing-e2e-autores-43fdfe-6bbcb85f7-wv7k8\" (UID: \"80a7d765-9b80-49f8-8d64-f72ec2acb5d7\") " pod="openshift-ingress/payload-processing-e2e-autores-43fdfe-6bbcb85f7-wv7k8" Sep 18 06:40:40.851919 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:40.851877 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knk25\" (UniqueName: \"kubernetes.io/projected/80a7d765-9b80-49f8-8d64-f72ec2acb5d7-kube-api-access-knk25\") pod \"payload-processing-e2e-autores-43fdfe-6bbcb85f7-wv7k8\" (UID: \"80a7d765-9b80-49f8-8d64-f72ec2acb5d7\") " pod="openshift-ingress/payload-processing-e2e-autores-43fdfe-6bbcb85f7-wv7k8" Sep 18 06:40:40.852264 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:40.852240 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80a7d765-9b80-49f8-8d64-f72ec2acb5d7-plugins-config-volume\") pod \"payload-processing-e2e-autores-43fdfe-6bbcb85f7-wv7k8\" (UID: \"80a7d765-9b80-49f8-8d64-f72ec2acb5d7\") " pod="openshift-ingress/payload-processing-e2e-autores-43fdfe-6bbcb85f7-wv7k8" Sep 18 06:40:40.853742 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:40.853717 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80a7d765-9b80-49f8-8d64-f72ec2acb5d7-plugins-config-volume\") pod \"payload-processing-e2e-autores-43fdfe-6bbcb85f7-wv7k8\" (UID: \"80a7d765-9b80-49f8-8d64-f72ec2acb5d7\") " pod="openshift-ingress/payload-processing-e2e-autores-43fdfe-6bbcb85f7-wv7k8" Sep 18 06:40:40.866470 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:40.866446 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knk25\" (UniqueName: \"kubernetes.io/projected/80a7d765-9b80-49f8-8d64-f72ec2acb5d7-kube-api-access-knk25\") pod \"payload-processing-e2e-autores-43fdfe-6bbcb85f7-wv7k8\" (UID: \"80a7d765-9b80-49f8-8d64-f72ec2acb5d7\") " pod="openshift-ingress/payload-processing-e2e-autores-43fdfe-6bbcb85f7-wv7k8" Sep 18 06:40:41.040951 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:41.040871 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-43fdfe-6bbcb85f7-wv7k8" Sep 18 06:40:41.327447 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:41.327324 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-654463-db8fb8547-x6klj" Sep 18 06:40:41.420995 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:41.420964 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-43fdfe-6bbcb85f7-wv7k8"] Sep 18 06:40:42.334847 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:42.334820 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-946609-67f4478978-fmbp7" Sep 18 06:40:42.335011 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:42.334906 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-946609-9fd88897f-68ljl" Sep 18 06:40:42.474443 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:42.454113 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-43fdfe-6bbcb85f7-wv7k8" Sep 18 06:40:42.475030 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:42.474962 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-43fdfe-6bbcb85f7-wv7k8" event={"ID":"80a7d765-9b80-49f8-8d64-f72ec2acb5d7","Type":"ContainerStarted","Data":"0f6624657c7ba926e2913082c5905d8b5d88737d56d578492fff53bb4ef86381"} Sep 18 06:40:42.475212 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:42.475193 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-43fdfe-6bbcb85f7-wv7k8" event={"ID":"80a7d765-9b80-49f8-8d64-f72ec2acb5d7","Type":"ContainerStarted","Data":"f17ed739bc569d33cdc437e760dd7b9e5c57ed6dca1b1e210e9b1e733a2590ed"} Sep 18 06:40:42.478259 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:42.478208 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-43fdfe-6bbcb85f7-wv7k8" podStartSLOduration=2.478190776 podStartE2EDuration="2.478190776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:40:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:40:42.477489348 +0000 UTC m=+1104.216905542" watchObservedRunningTime="2026-09-18 06:40:42.478190776 +0000 UTC m=+1104.217606952" Sep 18 06:40:46.472084 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:46.472049 2567 generic.go:356] "Generic (PLEG): container finished" podID="504bef64-f0e4-4856-a251-9296702689c4" containerID="280d89efcbcbad5984a021c0fac52bcf28d945b59cb8c97b0494623e0cdc1835" exitCode=0 Sep 18 06:40:46.472504 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:46.472113 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw" event={"ID":"504bef64-f0e4-4856-a251-9296702689c4","Type":"ContainerDied","Data":"280d89efcbcbad5984a021c0fac52bcf28d945b59cb8c97b0494623e0cdc1835"} Sep 18 06:40:46.472648 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:46.472574 2567 scope.go:117] "RemoveContainer" containerID="280d89efcbcbad5984a021c0fac52bcf28d945b59cb8c97b0494623e0cdc1835" Sep 18 06:40:46.474197 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:46.473947 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:40:46.474197 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:46.474028 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:40:46.474197 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:46.474070 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:40:46.474468 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:46.474405 2567 generic.go:356] "Generic (PLEG): container finished" podID="5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8" containerID="d45ebdf186426c3e3c1691771b762497d85f1643093e5d0f4fc83446874b6dca" exitCode=0 Sep 18 06:40:46.474529 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:46.474468 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262" event={"ID":"5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8","Type":"ContainerDied","Data":"d45ebdf186426c3e3c1691771b762497d85f1643093e5d0f4fc83446874b6dca"} Sep 18 06:40:46.474881 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:46.474860 2567 scope.go:117] "RemoveContainer" containerID="d45ebdf186426c3e3c1691771b762497d85f1643093e5d0f4fc83446874b6dca" Sep 18 06:40:46.476090 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:46.476067 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:40:46.476152 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:46.476129 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:40:46.476188 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:46.476162 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:40:47.484157 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:47.484125 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw" event={"ID":"504bef64-f0e4-4856-a251-9296702689c4","Type":"ContainerStarted","Data":"06caf4dd97134a797990b3927b1a105051a8dda775fc8fe29cc4456cb6f943b0"} Sep 18 06:40:47.486310 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:47.486286 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262" event={"ID":"5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8","Type":"ContainerStarted","Data":"28c51a24335d3d5229316c14ae77f3a0ab67bb80a9ba4571e26fd9646dfe60ab"} Sep 18 06:40:47.617178 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:47.617149 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262" Sep 18 06:40:47.617351 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:47.617328 2567 patch_prober.go:28] interesting pod/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Sep 18 06:40:47.617428 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:47.617408 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262" podUID="5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Sep 18 06:40:47.686470 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:47.686426 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw" Sep 18 06:40:47.686761 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:47.686730 2567 patch_prober.go:28] interesting pod/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Sep 18 06:40:47.686880 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:47.686789 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw" podUID="504bef64-f0e4-4856-a251-9296702689c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Sep 18 06:40:48.617225 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:48.617189 2567 patch_prober.go:28] interesting pod/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Sep 18 06:40:48.617649 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:48.617246 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262" podUID="5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Sep 18 06:40:48.687340 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:48.687295 2567 patch_prober.go:28] interesting pod/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Sep 18 06:40:48.687570 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:48.687384 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw" podUID="504bef64-f0e4-4856-a251-9296702689c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Sep 18 06:40:49.617815 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:49.617778 2567 patch_prober.go:28] interesting pod/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Sep 18 06:40:49.618176 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:49.617824 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262" podUID="5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Sep 18 06:40:49.687164 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:49.687122 2567 patch_prober.go:28] interesting pod/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Sep 18 06:40:49.687390 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:49.687190 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw" podUID="504bef64-f0e4-4856-a251-9296702689c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Sep 18 06:40:50.617810 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:50.617778 2567 patch_prober.go:28] interesting pod/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Sep 18 06:40:50.617994 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:50.617828 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262" podUID="5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Sep 18 06:40:50.686968 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:50.686929 2567 patch_prober.go:28] interesting pod/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Sep 18 06:40:50.687159 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:50.686988 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw" podUID="504bef64-f0e4-4856-a251-9296702689c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Sep 18 06:40:50.843921 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:50.843885 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0da33e9c-bmznm"] Sep 18 06:40:50.849920 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:50.849889 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0da33e9c-bmznm" Sep 18 06:40:50.853893 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:50.852480 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-88t7h\"" Sep 18 06:40:50.853893 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:50.852739 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 18 06:40:50.854076 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:50.854034 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0da33e9c-bmznm"] Sep 18 06:40:51.015101 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:51.015009 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04825b6c-ed4b-47fa-985a-8445d78100bf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0da33e9c-bmznm\" (UID: \"04825b6c-ed4b-47fa-985a-8445d78100bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0da33e9c-bmznm" Sep 18 06:40:51.116194 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:51.116158 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04825b6c-ed4b-47fa-985a-8445d78100bf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0da33e9c-bmznm\" (UID: \"04825b6c-ed4b-47fa-985a-8445d78100bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0da33e9c-bmznm" Sep 18 06:40:51.116995 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:51.116973 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04825b6c-ed4b-47fa-985a-8445d78100bf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0da33e9c-bmznm\" (UID: \"04825b6c-ed4b-47fa-985a-8445d78100bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0da33e9c-bmznm" Sep 18 06:40:51.166537 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:51.166502 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0da33e9c-bmznm" Sep 18 06:40:51.439194 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:51.439166 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-0da33e9c-68795bc8cf-qqsg9" Sep 18 06:40:51.500147 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:51.500115 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0da33e9c-bmznm"] Sep 18 06:40:51.520997 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:40:51.520928 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04825b6c_ed4b_47fa_985a_8445d78100bf.slice/crio-da7045bf44a3e65cc5f1ab0f4e3d9a97d915e62ee7772d4b115c0f81b979e2ba WatchSource:0}: Error finding container da7045bf44a3e65cc5f1ab0f4e3d9a97d915e62ee7772d4b115c0f81b979e2ba: Status 404 returned error can't find the container with id da7045bf44a3e65cc5f1ab0f4e3d9a97d915e62ee7772d4b115c0f81b979e2ba Sep 18 06:40:51.523325 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:51.523304 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 18 06:40:51.617670 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:51.617630 2567 patch_prober.go:28] interesting pod/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Sep 18 06:40:51.617866 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:51.617691 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262" podUID="5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Sep 18 06:40:51.686901 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:51.686857 2567 patch_prober.go:28] interesting pod/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Sep 18 06:40:51.687420 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:51.686927 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw" podUID="504bef64-f0e4-4856-a251-9296702689c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Sep 18 06:40:52.448614 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:52.448577 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0da33e9c-6974b99bfd-cvqm8" Sep 18 06:40:52.532302 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:52.532261 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0da33e9c-bmznm" event={"ID":"04825b6c-ed4b-47fa-985a-8445d78100bf","Type":"ContainerStarted","Data":"da7045bf44a3e65cc5f1ab0f4e3d9a97d915e62ee7772d4b115c0f81b979e2ba"} Sep 18 06:40:52.617699 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:52.617650 2567 patch_prober.go:28] interesting pod/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Sep 18 06:40:52.617898 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:52.617706 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262" podUID="5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Sep 18 06:40:52.688390 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:52.687490 2567 patch_prober.go:28] interesting pod/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Sep 18 06:40:52.688390 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:52.687560 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw" podUID="504bef64-f0e4-4856-a251-9296702689c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Sep 18 06:40:53.617888 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:53.617849 2567 patch_prober.go:28] interesting pod/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Sep 18 06:40:53.618059 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:53.617910 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262" podUID="5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Sep 18 06:40:53.691669 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:53.691635 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw" Sep 18 06:40:53.692126 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:53.692032 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw" Sep 18 06:40:53.693029 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:53.693007 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw" Sep 18 06:40:54.459557 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:54.459530 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-43fdfe-6bbcb85f7-wv7k8" Sep 18 06:40:54.622227 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:54.622195 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262" Sep 18 06:40:54.622659 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:54.622626 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262" Sep 18 06:40:54.623338 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:54.623319 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262" Sep 18 06:40:54.686608 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:54.686577 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-708b6f-654cf9f9db-mfx72"] Sep 18 06:40:54.696415 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:54.696360 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-708b6f-654cf9f9db-mfx72" Sep 18 06:40:54.701125 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:54.701085 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-708b6f\"" Sep 18 06:40:54.701680 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:54.701629 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-708b6f-dockercfg-84fpd\"" Sep 18 06:40:54.704948 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:54.704925 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-708b6f-654cf9f9db-mfx72"] Sep 18 06:40:54.753044 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:54.752976 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1caa877a-113d-4a99-a443-c9374528c9eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-708b6f-654cf9f9db-mfx72\" (UID: \"1caa877a-113d-4a99-a443-c9374528c9eb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-708b6f-654cf9f9db-mfx72" Sep 18 06:40:54.753044 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:54.753039 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjflc\" (UniqueName: \"kubernetes.io/projected/1caa877a-113d-4a99-a443-c9374528c9eb-kube-api-access-xjflc\") pod \"payload-pre-processing-e2e-inf-b-708b6f-654cf9f9db-mfx72\" (UID: \"1caa877a-113d-4a99-a443-c9374528c9eb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-708b6f-654cf9f9db-mfx72" Sep 18 06:40:54.854271 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:54.853972 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjflc\" (UniqueName: \"kubernetes.io/projected/1caa877a-113d-4a99-a443-c9374528c9eb-kube-api-access-xjflc\") pod \"payload-pre-processing-e2e-inf-b-708b6f-654cf9f9db-mfx72\" (UID: \"1caa877a-113d-4a99-a443-c9374528c9eb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-708b6f-654cf9f9db-mfx72" Sep 18 06:40:54.854271 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:54.854130 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1caa877a-113d-4a99-a443-c9374528c9eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-708b6f-654cf9f9db-mfx72\" (UID: \"1caa877a-113d-4a99-a443-c9374528c9eb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-708b6f-654cf9f9db-mfx72" Sep 18 06:40:54.854950 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:54.854925 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1caa877a-113d-4a99-a443-c9374528c9eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-708b6f-654cf9f9db-mfx72\" (UID: \"1caa877a-113d-4a99-a443-c9374528c9eb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-708b6f-654cf9f9db-mfx72" Sep 18 06:40:54.862209 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:54.862186 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjflc\" (UniqueName: \"kubernetes.io/projected/1caa877a-113d-4a99-a443-c9374528c9eb-kube-api-access-xjflc\") pod \"payload-pre-processing-e2e-inf-b-708b6f-654cf9f9db-mfx72\" (UID: \"1caa877a-113d-4a99-a443-c9374528c9eb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-708b6f-654cf9f9db-mfx72" Sep 18 06:40:55.014976 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:55.014892 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-708b6f-654cf9f9db-mfx72" Sep 18 06:40:55.350337 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:55.350310 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-708b6f-654cf9f9db-mfx72"] Sep 18 06:40:55.359357 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:40:55.359327 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1caa877a_113d_4a99_a443_c9374528c9eb.slice/crio-20ea43693829bf348a0282e2f9da93d6551e486b5b664f22b61e4384427ae14d WatchSource:0}: Error finding container 20ea43693829bf348a0282e2f9da93d6551e486b5b664f22b61e4384427ae14d: Status 404 returned error can't find the container with id 20ea43693829bf348a0282e2f9da93d6551e486b5b664f22b61e4384427ae14d Sep 18 06:40:55.556896 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:55.556802 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-708b6f-654cf9f9db-mfx72" event={"ID":"1caa877a-113d-4a99-a443-c9374528c9eb","Type":"ContainerStarted","Data":"d4f9d526112b4dc742db8c122609bfd485820cafea38c7e1d12b18e46fb83bb4"} Sep 18 06:40:55.556896 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:55.556874 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-708b6f-654cf9f9db-mfx72" Sep 18 06:40:55.556896 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:55.556898 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-708b6f-654cf9f9db-mfx72" event={"ID":"1caa877a-113d-4a99-a443-c9374528c9eb","Type":"ContainerStarted","Data":"20ea43693829bf348a0282e2f9da93d6551e486b5b664f22b61e4384427ae14d"} Sep 18 06:40:55.559852 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:55.559810 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0da33e9c-bmznm" event={"ID":"04825b6c-ed4b-47fa-985a-8445d78100bf","Type":"ContainerStarted","Data":"e10826b7926dc4af2fc1d6f23d547b3e6fb0815ebddb9e06ec9b2f89a8d8e20b"} Sep 18 06:40:55.574624 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:55.574569 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-708b6f-654cf9f9db-mfx72" podStartSLOduration=1.574548771 podStartE2EDuration="1.574548771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:40:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:40:55.573093322 +0000 UTC m=+1117.312509661" watchObservedRunningTime="2026-09-18 06:40:55.574548771 +0000 UTC m=+1117.313964946" Sep 18 06:40:55.600462 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:55.600328 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0da33e9c-bmznm" podStartSLOduration=1.970645367 podStartE2EDuration="5.600311026s" totalImagesPullingTime="3.629665659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:40:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:40:55.587185983 +0000 UTC m=+1117.326602158" watchObservedRunningTime="2026-09-18 06:40:55.600311026 +0000 UTC m=+1117.339727200" Sep 18 06:40:56.567664 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:56.567629 2567 generic.go:356] "Generic (PLEG): container finished" podID="04825b6c-ed4b-47fa-985a-8445d78100bf" containerID="e10826b7926dc4af2fc1d6f23d547b3e6fb0815ebddb9e06ec9b2f89a8d8e20b" exitCode=0 Sep 18 06:40:56.568068 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:56.567715 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0da33e9c-bmznm" event={"ID":"04825b6c-ed4b-47fa-985a-8445d78100bf","Type":"ContainerDied","Data":"e10826b7926dc4af2fc1d6f23d547b3e6fb0815ebddb9e06ec9b2f89a8d8e20b"} Sep 18 06:40:57.718013 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:57.717986 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0da33e9c-bmznm" Sep 18 06:40:57.782939 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:57.782914 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04825b6c-ed4b-47fa-985a-8445d78100bf-maas-api-service-ca\") pod \"04825b6c-ed4b-47fa-985a-8445d78100bf\" (UID: \"04825b6c-ed4b-47fa-985a-8445d78100bf\") " Sep 18 06:40:57.783268 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:57.783243 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04825b6c-ed4b-47fa-985a-8445d78100bf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "04825b6c-ed4b-47fa-985a-8445d78100bf" (UID: "04825b6c-ed4b-47fa-985a-8445d78100bf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 06:40:57.883905 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:57.883881 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04825b6c-ed4b-47fa-985a-8445d78100bf-maas-api-service-ca\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:40:58.579727 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:58.579702 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0da33e9c-bmznm" Sep 18 06:40:58.579870 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:58.579746 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0da33e9c-bmznm" event={"ID":"04825b6c-ed4b-47fa-985a-8445d78100bf","Type":"ContainerDied","Data":"da7045bf44a3e65cc5f1ab0f4e3d9a97d915e62ee7772d4b115c0f81b979e2ba"} Sep 18 06:40:58.579870 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:40:58.579794 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da7045bf44a3e65cc5f1ab0f4e3d9a97d915e62ee7772d4b115c0f81b979e2ba" Sep 18 06:41:01.532317 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:01.532283 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0da33e9c-68795bc8cf-qqsg9"] Sep 18 06:41:01.533107 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:01.533075 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-0da33e9c-68795bc8cf-qqsg9" podUID="ad0ef75d-4d0f-4295-9890-b14f9e55dc54" containerName="payload-processing" containerID="cri-o://0aecd3947dda63b7b3ea961fe7393d7a46cc192d54edfeffdf837b1086fc2a1c" gracePeriod=30 Sep 18 06:41:01.536819 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:01.536793 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0da33e9c-6974b99bfd-cvqm8"] Sep 18 06:41:01.537608 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:01.537183 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-0da33e9c-6974b99bfd-cvqm8" podUID="a2f576ba-7293-4e5f-8cbf-e346ddf82310" containerName="payload-pre-processing" containerID="cri-o://1e34ea23fa97026c477c21290acd875e51a3b6db60a454d11d1b135fafc6c6ca" gracePeriod=30 Sep 18 06:41:01.679161 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:01.679117 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0da33e9c-bmznm"] Sep 18 06:41:01.681338 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:01.681312 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0da33e9c-bmznm"] Sep 18 06:41:01.854469 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:01.854449 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0da33e9c-68795bc8cf-qqsg9" Sep 18 06:41:01.857463 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:01.857441 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0da33e9c-6974b99bfd-cvqm8" Sep 18 06:41:01.920917 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:01.920887 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad0ef75d-4d0f-4295-9890-b14f9e55dc54-plugins-config-volume\") pod \"ad0ef75d-4d0f-4295-9890-b14f9e55dc54\" (UID: \"ad0ef75d-4d0f-4295-9890-b14f9e55dc54\") " Sep 18 06:41:01.921062 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:01.920925 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2f576ba-7293-4e5f-8cbf-e346ddf82310-plugins-config-volume\") pod \"a2f576ba-7293-4e5f-8cbf-e346ddf82310\" (UID: \"a2f576ba-7293-4e5f-8cbf-e346ddf82310\") " Sep 18 06:41:01.921062 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:01.921017 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xwrbd\" (UniqueName: \"kubernetes.io/projected/a2f576ba-7293-4e5f-8cbf-e346ddf82310-kube-api-access-xwrbd\") pod \"a2f576ba-7293-4e5f-8cbf-e346ddf82310\" (UID: \"a2f576ba-7293-4e5f-8cbf-e346ddf82310\") " Sep 18 06:41:01.921235 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:01.921217 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x8kfz\" (UniqueName: \"kubernetes.io/projected/ad0ef75d-4d0f-4295-9890-b14f9e55dc54-kube-api-access-x8kfz\") pod \"ad0ef75d-4d0f-4295-9890-b14f9e55dc54\" (UID: \"ad0ef75d-4d0f-4295-9890-b14f9e55dc54\") " Sep 18 06:41:01.921643 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:01.921317 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad0ef75d-4d0f-4295-9890-b14f9e55dc54-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ad0ef75d-4d0f-4295-9890-b14f9e55dc54" (UID: "ad0ef75d-4d0f-4295-9890-b14f9e55dc54"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 06:41:01.921643 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:01.921330 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2f576ba-7293-4e5f-8cbf-e346ddf82310-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a2f576ba-7293-4e5f-8cbf-e346ddf82310" (UID: "a2f576ba-7293-4e5f-8cbf-e346ddf82310"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 06:41:01.922024 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:01.921982 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad0ef75d-4d0f-4295-9890-b14f9e55dc54-plugins-config-volume\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:41:01.922024 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:01.922010 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2f576ba-7293-4e5f-8cbf-e346ddf82310-plugins-config-volume\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:41:01.924229 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:01.924201 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad0ef75d-4d0f-4295-9890-b14f9e55dc54-kube-api-access-x8kfz" (OuterVolumeSpecName: "kube-api-access-x8kfz") pod "ad0ef75d-4d0f-4295-9890-b14f9e55dc54" (UID: "ad0ef75d-4d0f-4295-9890-b14f9e55dc54"). InnerVolumeSpecName "kube-api-access-x8kfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 06:41:01.924732 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:01.924597 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2f576ba-7293-4e5f-8cbf-e346ddf82310-kube-api-access-xwrbd" (OuterVolumeSpecName: "kube-api-access-xwrbd") pod "a2f576ba-7293-4e5f-8cbf-e346ddf82310" (UID: "a2f576ba-7293-4e5f-8cbf-e346ddf82310"). InnerVolumeSpecName "kube-api-access-xwrbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 06:41:02.022524 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:02.022500 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8kfz\" (UniqueName: \"kubernetes.io/projected/ad0ef75d-4d0f-4295-9890-b14f9e55dc54-kube-api-access-x8kfz\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:41:02.022524 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:02.022521 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwrbd\" (UniqueName: \"kubernetes.io/projected/a2f576ba-7293-4e5f-8cbf-e346ddf82310-kube-api-access-xwrbd\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:41:02.446116 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:02.446080 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04825b6c-ed4b-47fa-985a-8445d78100bf" path="/var/lib/kubelet/pods/04825b6c-ed4b-47fa-985a-8445d78100bf/volumes" Sep 18 06:41:02.604152 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:02.604124 2567 generic.go:356] "Generic (PLEG): container finished" podID="ad0ef75d-4d0f-4295-9890-b14f9e55dc54" containerID="0aecd3947dda63b7b3ea961fe7393d7a46cc192d54edfeffdf837b1086fc2a1c" exitCode=0 Sep 18 06:41:02.604573 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:02.604167 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0da33e9c-68795bc8cf-qqsg9" Sep 18 06:41:02.604573 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:02.604208 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0da33e9c-68795bc8cf-qqsg9" event={"ID":"ad0ef75d-4d0f-4295-9890-b14f9e55dc54","Type":"ContainerDied","Data":"0aecd3947dda63b7b3ea961fe7393d7a46cc192d54edfeffdf837b1086fc2a1c"} Sep 18 06:41:02.604573 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:02.604258 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0da33e9c-68795bc8cf-qqsg9" event={"ID":"ad0ef75d-4d0f-4295-9890-b14f9e55dc54","Type":"ContainerDied","Data":"e8742c3db38cb11dcb049024d2055ba68fc9ee699937abf7da03fb2075986890"} Sep 18 06:41:02.604573 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:02.604281 2567 scope.go:117] "RemoveContainer" containerID="0aecd3947dda63b7b3ea961fe7393d7a46cc192d54edfeffdf837b1086fc2a1c" Sep 18 06:41:02.605769 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:02.605748 2567 generic.go:356] "Generic (PLEG): container finished" podID="a2f576ba-7293-4e5f-8cbf-e346ddf82310" containerID="1e34ea23fa97026c477c21290acd875e51a3b6db60a454d11d1b135fafc6c6ca" exitCode=0 Sep 18 06:41:02.605857 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:02.605785 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0da33e9c-6974b99bfd-cvqm8" Sep 18 06:41:02.605918 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:02.605843 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0da33e9c-6974b99bfd-cvqm8" event={"ID":"a2f576ba-7293-4e5f-8cbf-e346ddf82310","Type":"ContainerDied","Data":"1e34ea23fa97026c477c21290acd875e51a3b6db60a454d11d1b135fafc6c6ca"} Sep 18 06:41:02.605918 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:02.605897 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0da33e9c-6974b99bfd-cvqm8" event={"ID":"a2f576ba-7293-4e5f-8cbf-e346ddf82310","Type":"ContainerDied","Data":"381a5ebb332baa9f0fef38f6f5e958c8085292a3afc6219996c0755cd6dbb3bc"} Sep 18 06:41:02.614612 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:02.614596 2567 scope.go:117] "RemoveContainer" containerID="0aecd3947dda63b7b3ea961fe7393d7a46cc192d54edfeffdf837b1086fc2a1c" Sep 18 06:41:02.614858 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:41:02.614844 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0aecd3947dda63b7b3ea961fe7393d7a46cc192d54edfeffdf837b1086fc2a1c\": container with ID starting with 0aecd3947dda63b7b3ea961fe7393d7a46cc192d54edfeffdf837b1086fc2a1c not found: ID does not exist" containerID="0aecd3947dda63b7b3ea961fe7393d7a46cc192d54edfeffdf837b1086fc2a1c" Sep 18 06:41:02.614900 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:02.614865 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0aecd3947dda63b7b3ea961fe7393d7a46cc192d54edfeffdf837b1086fc2a1c"} err="failed to get container status \"0aecd3947dda63b7b3ea961fe7393d7a46cc192d54edfeffdf837b1086fc2a1c\": rpc error: code = NotFound desc = could not find container \"0aecd3947dda63b7b3ea961fe7393d7a46cc192d54edfeffdf837b1086fc2a1c\": container with ID starting with 0aecd3947dda63b7b3ea961fe7393d7a46cc192d54edfeffdf837b1086fc2a1c not found: ID does not exist" Sep 18 06:41:02.614900 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:02.614879 2567 scope.go:117] "RemoveContainer" containerID="1e34ea23fa97026c477c21290acd875e51a3b6db60a454d11d1b135fafc6c6ca" Sep 18 06:41:02.623792 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:02.623767 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0da33e9c-6974b99bfd-cvqm8"] Sep 18 06:41:02.624813 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:02.624795 2567 scope.go:117] "RemoveContainer" containerID="1e34ea23fa97026c477c21290acd875e51a3b6db60a454d11d1b135fafc6c6ca" Sep 18 06:41:02.625074 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:41:02.625057 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e34ea23fa97026c477c21290acd875e51a3b6db60a454d11d1b135fafc6c6ca\": container with ID starting with 1e34ea23fa97026c477c21290acd875e51a3b6db60a454d11d1b135fafc6c6ca not found: ID does not exist" containerID="1e34ea23fa97026c477c21290acd875e51a3b6db60a454d11d1b135fafc6c6ca" Sep 18 06:41:02.625125 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:02.625080 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e34ea23fa97026c477c21290acd875e51a3b6db60a454d11d1b135fafc6c6ca"} err="failed to get container status \"1e34ea23fa97026c477c21290acd875e51a3b6db60a454d11d1b135fafc6c6ca\": rpc error: code = NotFound desc = could not find container \"1e34ea23fa97026c477c21290acd875e51a3b6db60a454d11d1b135fafc6c6ca\": container with ID starting with 1e34ea23fa97026c477c21290acd875e51a3b6db60a454d11d1b135fafc6c6ca not found: ID does not exist" Sep 18 06:41:02.627143 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:02.627123 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0da33e9c-6974b99bfd-cvqm8"] Sep 18 06:41:02.638763 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:02.638740 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0da33e9c-68795bc8cf-qqsg9"] Sep 18 06:41:02.644452 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:02.644432 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0da33e9c-68795bc8cf-qqsg9"] Sep 18 06:41:02.716199 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:02.716122 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg"] Sep 18 06:41:02.716651 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:02.716629 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad0ef75d-4d0f-4295-9890-b14f9e55dc54" containerName="payload-processing" Sep 18 06:41:02.716651 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:02.716650 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad0ef75d-4d0f-4295-9890-b14f9e55dc54" containerName="payload-processing" Sep 18 06:41:02.716827 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:02.716660 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04825b6c-ed4b-47fa-985a-8445d78100bf" containerName="revoke-keys" Sep 18 06:41:02.716827 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:02.716666 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="04825b6c-ed4b-47fa-985a-8445d78100bf" containerName="revoke-keys" Sep 18 06:41:02.716827 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:02.716688 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2f576ba-7293-4e5f-8cbf-e346ddf82310" containerName="payload-pre-processing" Sep 18 06:41:02.716827 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:02.716693 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2f576ba-7293-4e5f-8cbf-e346ddf82310" containerName="payload-pre-processing" Sep 18 06:41:02.716827 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:02.716770 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2f576ba-7293-4e5f-8cbf-e346ddf82310" containerName="payload-pre-processing" Sep 18 06:41:02.716827 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:02.716781 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad0ef75d-4d0f-4295-9890-b14f9e55dc54" containerName="payload-processing" Sep 18 06:41:02.716827 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:02.716791 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="04825b6c-ed4b-47fa-985a-8445d78100bf" containerName="revoke-keys" Sep 18 06:41:02.721492 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:02.721468 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg" Sep 18 06:41:02.724684 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:02.724663 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-994a4fd1-gw-openshift-default-dockercfg-2v5qk\"" Sep 18 06:41:02.731940 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:02.731922 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg"] Sep 18 06:41:02.831105 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:02.831071 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfc46a52-5b82-4f2a-8b32-3af12d8270b1-istio-data\") pod \"e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg\" (UID: \"cfc46a52-5b82-4f2a-8b32-3af12d8270b1\") " pod="openshift-ingress/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg" Sep 18 06:41:02.831296 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:02.831136 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfc46a52-5b82-4f2a-8b32-3af12d8270b1-istio-token\") pod \"e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg\" (UID: \"cfc46a52-5b82-4f2a-8b32-3af12d8270b1\") " pod="openshift-ingress/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg" Sep 18 06:41:02.831296 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:02.831211 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfc46a52-5b82-4f2a-8b32-3af12d8270b1-istiod-ca-cert\") pod \"e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg\" (UID: \"cfc46a52-5b82-4f2a-8b32-3af12d8270b1\") " pod="openshift-ingress/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg" Sep 18 06:41:02.831296 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:02.831240 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6hpl\" (UniqueName: \"kubernetes.io/projected/cfc46a52-5b82-4f2a-8b32-3af12d8270b1-kube-api-access-q6hpl\") pod \"e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg\" (UID: \"cfc46a52-5b82-4f2a-8b32-3af12d8270b1\") " pod="openshift-ingress/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg" Sep 18 06:41:02.831296 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:02.831286 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc46a52-5b82-4f2a-8b32-3af12d8270b1-workload-socket\") pod \"e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg\" (UID: \"cfc46a52-5b82-4f2a-8b32-3af12d8270b1\") " pod="openshift-ingress/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg" Sep 18 06:41:02.831545 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:02.831321 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfc46a52-5b82-4f2a-8b32-3af12d8270b1-workload-certs\") pod \"e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg\" (UID: \"cfc46a52-5b82-4f2a-8b32-3af12d8270b1\") " pod="openshift-ingress/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg" Sep 18 06:41:02.831545 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:02.831343 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfc46a52-5b82-4f2a-8b32-3af12d8270b1-istio-envoy\") pod \"e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg\" (UID: \"cfc46a52-5b82-4f2a-8b32-3af12d8270b1\") " pod="openshift-ingress/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg" Sep 18 06:41:02.831545 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:02.831382 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc46a52-5b82-4f2a-8b32-3af12d8270b1-credential-socket\") pod \"e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg\" (UID: \"cfc46a52-5b82-4f2a-8b32-3af12d8270b1\") " pod="openshift-ingress/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg" Sep 18 06:41:02.831545 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:02.831409 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfc46a52-5b82-4f2a-8b32-3af12d8270b1-istio-podinfo\") pod \"e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg\" (UID: \"cfc46a52-5b82-4f2a-8b32-3af12d8270b1\") " pod="openshift-ingress/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg" Sep 18 06:41:02.932176 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:02.932147 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc46a52-5b82-4f2a-8b32-3af12d8270b1-workload-socket\") pod \"e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg\" (UID: \"cfc46a52-5b82-4f2a-8b32-3af12d8270b1\") " pod="openshift-ingress/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg" Sep 18 06:41:02.932350 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:02.932200 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfc46a52-5b82-4f2a-8b32-3af12d8270b1-workload-certs\") pod \"e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg\" (UID: \"cfc46a52-5b82-4f2a-8b32-3af12d8270b1\") " pod="openshift-ingress/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg" Sep 18 06:41:02.932350 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:02.932232 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfc46a52-5b82-4f2a-8b32-3af12d8270b1-istio-envoy\") pod \"e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg\" (UID: \"cfc46a52-5b82-4f2a-8b32-3af12d8270b1\") " pod="openshift-ingress/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg" Sep 18 06:41:02.932350 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:02.932266 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc46a52-5b82-4f2a-8b32-3af12d8270b1-credential-socket\") pod \"e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg\" (UID: \"cfc46a52-5b82-4f2a-8b32-3af12d8270b1\") " pod="openshift-ingress/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg" Sep 18 06:41:02.932350 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:02.932295 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfc46a52-5b82-4f2a-8b32-3af12d8270b1-istio-podinfo\") pod \"e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg\" (UID: \"cfc46a52-5b82-4f2a-8b32-3af12d8270b1\") " pod="openshift-ingress/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg" Sep 18 06:41:02.932350 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:02.932345 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfc46a52-5b82-4f2a-8b32-3af12d8270b1-istio-data\") pod \"e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg\" (UID: \"cfc46a52-5b82-4f2a-8b32-3af12d8270b1\") " pod="openshift-ingress/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg" Sep 18 06:41:02.932678 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:02.932448 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfc46a52-5b82-4f2a-8b32-3af12d8270b1-istio-token\") pod \"e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg\" (UID: \"cfc46a52-5b82-4f2a-8b32-3af12d8270b1\") " pod="openshift-ingress/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg" Sep 18 06:41:02.932678 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:02.932493 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfc46a52-5b82-4f2a-8b32-3af12d8270b1-istiod-ca-cert\") pod \"e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg\" (UID: \"cfc46a52-5b82-4f2a-8b32-3af12d8270b1\") " pod="openshift-ingress/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg" Sep 18 06:41:02.932678 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:02.932541 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6hpl\" (UniqueName: \"kubernetes.io/projected/cfc46a52-5b82-4f2a-8b32-3af12d8270b1-kube-api-access-q6hpl\") pod \"e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg\" (UID: \"cfc46a52-5b82-4f2a-8b32-3af12d8270b1\") " pod="openshift-ingress/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg" Sep 18 06:41:02.932678 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:02.932613 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc46a52-5b82-4f2a-8b32-3af12d8270b1-workload-socket\") pod \"e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg\" (UID: \"cfc46a52-5b82-4f2a-8b32-3af12d8270b1\") " pod="openshift-ingress/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg" Sep 18 06:41:02.932678 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:02.932652 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfc46a52-5b82-4f2a-8b32-3af12d8270b1-workload-certs\") pod \"e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg\" (UID: \"cfc46a52-5b82-4f2a-8b32-3af12d8270b1\") " pod="openshift-ingress/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg" Sep 18 06:41:02.932678 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:02.932665 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc46a52-5b82-4f2a-8b32-3af12d8270b1-credential-socket\") pod \"e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg\" (UID: \"cfc46a52-5b82-4f2a-8b32-3af12d8270b1\") " pod="openshift-ingress/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg" Sep 18 06:41:02.932958 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:02.932781 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfc46a52-5b82-4f2a-8b32-3af12d8270b1-istio-data\") pod \"e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg\" (UID: \"cfc46a52-5b82-4f2a-8b32-3af12d8270b1\") " pod="openshift-ingress/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg" Sep 18 06:41:02.933312 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:02.933286 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfc46a52-5b82-4f2a-8b32-3af12d8270b1-istiod-ca-cert\") pod \"e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg\" (UID: \"cfc46a52-5b82-4f2a-8b32-3af12d8270b1\") " pod="openshift-ingress/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg" Sep 18 06:41:02.934853 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:02.934834 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfc46a52-5b82-4f2a-8b32-3af12d8270b1-istio-podinfo\") pod \"e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg\" (UID: \"cfc46a52-5b82-4f2a-8b32-3af12d8270b1\") " pod="openshift-ingress/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg" Sep 18 06:41:02.934931 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:02.934866 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfc46a52-5b82-4f2a-8b32-3af12d8270b1-istio-envoy\") pod \"e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg\" (UID: \"cfc46a52-5b82-4f2a-8b32-3af12d8270b1\") " pod="openshift-ingress/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg" Sep 18 06:41:02.941778 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:02.941760 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfc46a52-5b82-4f2a-8b32-3af12d8270b1-istio-token\") pod \"e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg\" (UID: \"cfc46a52-5b82-4f2a-8b32-3af12d8270b1\") " pod="openshift-ingress/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg" Sep 18 06:41:02.941966 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:02.941949 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6hpl\" (UniqueName: \"kubernetes.io/projected/cfc46a52-5b82-4f2a-8b32-3af12d8270b1-kube-api-access-q6hpl\") pod \"e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg\" (UID: \"cfc46a52-5b82-4f2a-8b32-3af12d8270b1\") " pod="openshift-ingress/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg" Sep 18 06:41:03.034234 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:03.034150 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg" Sep 18 06:41:03.312620 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:03.312585 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg"] Sep 18 06:41:03.313760 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:41:03.313734 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfc46a52_5b82_4f2a_8b32_3af12d8270b1.slice/crio-e87fd73c662f97905cdeeb9fada9cf2cdc68473692d59f4029f3e9cec2cc4447 WatchSource:0}: Error finding container e87fd73c662f97905cdeeb9fada9cf2cdc68473692d59f4029f3e9cec2cc4447: Status 404 returned error can't find the container with id e87fd73c662f97905cdeeb9fada9cf2cdc68473692d59f4029f3e9cec2cc4447 Sep 18 06:41:03.316080 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:03.316047 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:41:03.316139 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:03.316113 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:41:03.316184 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:03.316139 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:41:03.615605 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:03.615572 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg" event={"ID":"cfc46a52-5b82-4f2a-8b32-3af12d8270b1","Type":"ContainerStarted","Data":"c876d8a975d222bb642ea75a71e4c76c0ee7fe667d53fba0f48749423dd773e4"} Sep 18 06:41:03.615963 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:03.615624 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg" event={"ID":"cfc46a52-5b82-4f2a-8b32-3af12d8270b1","Type":"ContainerStarted","Data":"e87fd73c662f97905cdeeb9fada9cf2cdc68473692d59f4029f3e9cec2cc4447"} Sep 18 06:41:03.636946 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:03.636892 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg" podStartSLOduration=1.636877474 podStartE2EDuration="1.636877474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:41:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:41:03.635022239 +0000 UTC m=+1125.374438434" watchObservedRunningTime="2026-09-18 06:41:03.636877474 +0000 UTC m=+1125.376293691" Sep 18 06:41:04.035233 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:04.035147 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg" Sep 18 06:41:04.040020 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:04.039997 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg" Sep 18 06:41:04.450518 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:04.450485 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2f576ba-7293-4e5f-8cbf-e346ddf82310" path="/var/lib/kubelet/pods/a2f576ba-7293-4e5f-8cbf-e346ddf82310/volumes" Sep 18 06:41:04.450902 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:04.450883 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad0ef75d-4d0f-4295-9890-b14f9e55dc54" path="/var/lib/kubelet/pods/ad0ef75d-4d0f-4295-9890-b14f9e55dc54/volumes" Sep 18 06:41:04.621770 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:04.621737 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg" Sep 18 06:41:04.622550 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:04.622534 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg" Sep 18 06:41:06.570050 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:06.570018 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-708b6f-654cf9f9db-mfx72" Sep 18 06:41:07.642994 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:07.642961 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-994a4fd1-679b5c994-xpcgk"] Sep 18 06:41:07.646345 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:07.646327 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-994a4fd1-679b5c994-xpcgk" Sep 18 06:41:07.648873 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:07.648853 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-994a4fd1\"" Sep 18 06:41:07.648993 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:07.648874 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-994a4fd1-dockercfg-mmr6r\"" Sep 18 06:41:07.654556 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:07.654533 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-994a4fd1-679b5c994-xpcgk"] Sep 18 06:41:07.777896 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:07.777862 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7138fe78-684d-4f77-9061-a570a9211c82-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-994a4fd1-679b5c994-xpcgk\" (UID: \"7138fe78-684d-4f77-9061-a570a9211c82\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-994a4fd1-679b5c994-xpcgk" Sep 18 06:41:07.778082 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:07.777991 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pdk9\" (UniqueName: \"kubernetes.io/projected/7138fe78-684d-4f77-9061-a570a9211c82-kube-api-access-6pdk9\") pod \"payload-pre-processing-e2e-migrate-994a4fd1-679b5c994-xpcgk\" (UID: \"7138fe78-684d-4f77-9061-a570a9211c82\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-994a4fd1-679b5c994-xpcgk" Sep 18 06:41:07.878606 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:07.878571 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pdk9\" (UniqueName: \"kubernetes.io/projected/7138fe78-684d-4f77-9061-a570a9211c82-kube-api-access-6pdk9\") pod \"payload-pre-processing-e2e-migrate-994a4fd1-679b5c994-xpcgk\" (UID: \"7138fe78-684d-4f77-9061-a570a9211c82\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-994a4fd1-679b5c994-xpcgk" Sep 18 06:41:07.878776 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:07.878669 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7138fe78-684d-4f77-9061-a570a9211c82-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-994a4fd1-679b5c994-xpcgk\" (UID: \"7138fe78-684d-4f77-9061-a570a9211c82\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-994a4fd1-679b5c994-xpcgk" Sep 18 06:41:07.879312 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:07.879292 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7138fe78-684d-4f77-9061-a570a9211c82-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-994a4fd1-679b5c994-xpcgk\" (UID: \"7138fe78-684d-4f77-9061-a570a9211c82\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-994a4fd1-679b5c994-xpcgk" Sep 18 06:41:07.886244 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:07.886218 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pdk9\" (UniqueName: \"kubernetes.io/projected/7138fe78-684d-4f77-9061-a570a9211c82-kube-api-access-6pdk9\") pod \"payload-pre-processing-e2e-migrate-994a4fd1-679b5c994-xpcgk\" (UID: \"7138fe78-684d-4f77-9061-a570a9211c82\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-994a4fd1-679b5c994-xpcgk" Sep 18 06:41:07.959825 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:07.959769 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-994a4fd1-679b5c994-xpcgk" Sep 18 06:41:08.234475 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:41:08.234444 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7138fe78_684d_4f77_9061_a570a9211c82.slice/crio-dd0e847bc8f7da65e4162c2fcc97fc92c1e017bd9465116a78093bfd85a7d5e5 WatchSource:0}: Error finding container dd0e847bc8f7da65e4162c2fcc97fc92c1e017bd9465116a78093bfd85a7d5e5: Status 404 returned error can't find the container with id dd0e847bc8f7da65e4162c2fcc97fc92c1e017bd9465116a78093bfd85a7d5e5 Sep 18 06:41:08.234646 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:08.234627 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-994a4fd1-679b5c994-xpcgk"] Sep 18 06:41:08.649941 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:08.649904 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-994a4fd1-679b5c994-xpcgk" event={"ID":"7138fe78-684d-4f77-9061-a570a9211c82","Type":"ContainerStarted","Data":"ad08d9df5f0f35ba5a678b90859b58d01b6350d729203cddfbe249d33e1a3b1e"} Sep 18 06:41:08.650397 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:08.649965 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-994a4fd1-679b5c994-xpcgk" event={"ID":"7138fe78-684d-4f77-9061-a570a9211c82","Type":"ContainerStarted","Data":"dd0e847bc8f7da65e4162c2fcc97fc92c1e017bd9465116a78093bfd85a7d5e5"} Sep 18 06:41:08.650397 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:08.650132 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-994a4fd1-679b5c994-xpcgk" Sep 18 06:41:08.668541 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:08.668499 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-994a4fd1-679b5c994-xpcgk" podStartSLOduration=1.6684855889999999 podStartE2EDuration="1.668485589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:41:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:41:08.665208576 +0000 UTC m=+1130.404624777" watchObservedRunningTime="2026-09-18 06:41:08.668485589 +0000 UTC m=+1130.407901763" Sep 18 06:41:19.657804 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:19.657773 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-994a4fd1-679b5c994-xpcgk" Sep 18 06:41:29.772461 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:29.772429 2567 generic.go:356] "Generic (PLEG): container finished" podID="cfc46a52-5b82-4f2a-8b32-3af12d8270b1" containerID="c876d8a975d222bb642ea75a71e4c76c0ee7fe667d53fba0f48749423dd773e4" exitCode=0 Sep 18 06:41:29.772876 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:29.772503 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg" event={"ID":"cfc46a52-5b82-4f2a-8b32-3af12d8270b1","Type":"ContainerDied","Data":"c876d8a975d222bb642ea75a71e4c76c0ee7fe667d53fba0f48749423dd773e4"} Sep 18 06:41:29.772919 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:29.772896 2567 scope.go:117] "RemoveContainer" containerID="c876d8a975d222bb642ea75a71e4c76c0ee7fe667d53fba0f48749423dd773e4" Sep 18 06:41:29.774031 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:29.774003 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:41:29.774098 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:29.774069 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:41:29.774137 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:29.774104 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:41:30.781101 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:30.781064 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg" event={"ID":"cfc46a52-5b82-4f2a-8b32-3af12d8270b1","Type":"ContainerStarted","Data":"7fe7cc6416d12461d032af2b77ebac3441a4f359adeb3b320439a03dd16895b1"} Sep 18 06:41:31.035247 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:31.035156 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg" Sep 18 06:41:31.036511 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:31.036482 2567 patch_prober.go:28] interesting pod/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Sep 18 06:41:31.036643 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:31.036536 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg" podUID="cfc46a52-5b82-4f2a-8b32-3af12d8270b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Sep 18 06:41:32.034817 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:32.034740 2567 patch_prober.go:28] interesting pod/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Sep 18 06:41:32.034817 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:32.034797 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg" podUID="cfc46a52-5b82-4f2a-8b32-3af12d8270b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Sep 18 06:41:33.034695 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:33.034661 2567 patch_prober.go:28] interesting pod/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Sep 18 06:41:33.034874 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:33.034724 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg" podUID="cfc46a52-5b82-4f2a-8b32-3af12d8270b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Sep 18 06:41:34.034770 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:34.034734 2567 patch_prober.go:28] interesting pod/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Sep 18 06:41:34.034943 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:34.034798 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg" podUID="cfc46a52-5b82-4f2a-8b32-3af12d8270b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Sep 18 06:41:34.622144 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:34.622111 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg" Sep 18 06:41:35.035482 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:35.035399 2567 patch_prober.go:28] interesting pod/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Sep 18 06:41:35.035482 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:35.035455 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg" podUID="cfc46a52-5b82-4f2a-8b32-3af12d8270b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Sep 18 06:41:35.255002 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:35.254964 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-994a4fd1-89nhc"] Sep 18 06:41:35.258744 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:35.258715 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-994a4fd1-89nhc" Sep 18 06:41:35.261060 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:35.261031 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 18 06:41:35.261289 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:35.261267 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-88t7h\"" Sep 18 06:41:35.281074 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:35.281048 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-994a4fd1-89nhc"] Sep 18 06:41:35.362865 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:35.362825 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fe6dc00-7ab0-441c-a90c-6a03c6442ead-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-994a4fd1-89nhc\" (UID: \"3fe6dc00-7ab0-441c-a90c-6a03c6442ead\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-994a4fd1-89nhc" Sep 18 06:41:35.464304 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:35.464264 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fe6dc00-7ab0-441c-a90c-6a03c6442ead-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-994a4fd1-89nhc\" (UID: \"3fe6dc00-7ab0-441c-a90c-6a03c6442ead\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-994a4fd1-89nhc" Sep 18 06:41:35.465216 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:35.465165 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fe6dc00-7ab0-441c-a90c-6a03c6442ead-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-994a4fd1-89nhc\" (UID: \"3fe6dc00-7ab0-441c-a90c-6a03c6442ead\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-994a4fd1-89nhc" Sep 18 06:41:35.589689 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:35.589646 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-994a4fd1-89nhc" Sep 18 06:41:35.861596 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:35.861505 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-994a4fd1-89nhc"] Sep 18 06:41:35.876652 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:41:35.876619 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fe6dc00_7ab0_441c_a90c_6a03c6442ead.slice/crio-df7eaa3e5cd017397781dc992e389abb0abb704ad418d7375e9c96092c930e6b WatchSource:0}: Error finding container df7eaa3e5cd017397781dc992e389abb0abb704ad418d7375e9c96092c930e6b: Status 404 returned error can't find the container with id df7eaa3e5cd017397781dc992e389abb0abb704ad418d7375e9c96092c930e6b Sep 18 06:41:36.034844 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:36.034804 2567 patch_prober.go:28] interesting pod/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Sep 18 06:41:36.035032 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:36.034864 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg" podUID="cfc46a52-5b82-4f2a-8b32-3af12d8270b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Sep 18 06:41:36.821641 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:36.821612 2567 generic.go:356] "Generic (PLEG): container finished" podID="3fe6dc00-7ab0-441c-a90c-6a03c6442ead" containerID="1563c54b6a271e2a107894cf9da42def38cd9fd9e34c45b5702c3d92a18ccfea" exitCode=0 Sep 18 06:41:36.822008 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:36.821694 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-994a4fd1-89nhc" event={"ID":"3fe6dc00-7ab0-441c-a90c-6a03c6442ead","Type":"ContainerDied","Data":"1563c54b6a271e2a107894cf9da42def38cd9fd9e34c45b5702c3d92a18ccfea"} Sep 18 06:41:36.822008 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:36.821750 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-994a4fd1-89nhc" event={"ID":"3fe6dc00-7ab0-441c-a90c-6a03c6442ead","Type":"ContainerStarted","Data":"df7eaa3e5cd017397781dc992e389abb0abb704ad418d7375e9c96092c930e6b"} Sep 18 06:41:37.039333 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:37.039246 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg" Sep 18 06:41:37.040096 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:37.040079 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg" Sep 18 06:41:37.982051 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:37.982020 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-994a4fd1-89nhc" Sep 18 06:41:38.001854 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:38.001828 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fe6dc00-7ab0-441c-a90c-6a03c6442ead-maas-api-service-ca\") pod \"3fe6dc00-7ab0-441c-a90c-6a03c6442ead\" (UID: \"3fe6dc00-7ab0-441c-a90c-6a03c6442ead\") " Sep 18 06:41:38.002216 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:38.002183 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fe6dc00-7ab0-441c-a90c-6a03c6442ead-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3fe6dc00-7ab0-441c-a90c-6a03c6442ead" (UID: "3fe6dc00-7ab0-441c-a90c-6a03c6442ead"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 06:41:38.002413 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:38.002396 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fe6dc00-7ab0-441c-a90c-6a03c6442ead-maas-api-service-ca\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:41:38.835307 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:38.835273 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-994a4fd1-89nhc" event={"ID":"3fe6dc00-7ab0-441c-a90c-6a03c6442ead","Type":"ContainerDied","Data":"df7eaa3e5cd017397781dc992e389abb0abb704ad418d7375e9c96092c930e6b"} Sep 18 06:41:38.835513 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:38.835321 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df7eaa3e5cd017397781dc992e389abb0abb704ad418d7375e9c96092c930e6b" Sep 18 06:41:38.835513 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:38.835320 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-994a4fd1-89nhc" Sep 18 06:41:47.092353 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:47.092316 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-994a4fd1-679b5c994-xpcgk"] Sep 18 06:41:47.092788 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:47.092583 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-994a4fd1-679b5c994-xpcgk" podUID="7138fe78-684d-4f77-9061-a570a9211c82" containerName="payload-pre-processing" containerID="cri-o://ad08d9df5f0f35ba5a678b90859b58d01b6350d729203cddfbe249d33e1a3b1e" gracePeriod=30 Sep 18 06:41:47.235588 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:47.235543 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-994a4fd1-89nhc"] Sep 18 06:41:47.238164 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:47.238141 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-994a4fd1-89nhc"] Sep 18 06:41:47.374163 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:47.374133 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg"] Sep 18 06:41:47.374418 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:47.374391 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg" podUID="cfc46a52-5b82-4f2a-8b32-3af12d8270b1" containerName="istio-proxy" containerID="cri-o://7fe7cc6416d12461d032af2b77ebac3441a4f359adeb3b320439a03dd16895b1" gracePeriod=30 Sep 18 06:41:47.390316 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:47.390288 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-994a4fd1-679b5c994-xpcgk" Sep 18 06:41:47.491792 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:47.491725 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7138fe78-684d-4f77-9061-a570a9211c82-plugins-config-volume\") pod \"7138fe78-684d-4f77-9061-a570a9211c82\" (UID: \"7138fe78-684d-4f77-9061-a570a9211c82\") " Sep 18 06:41:47.492499 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:47.492471 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7138fe78-684d-4f77-9061-a570a9211c82-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7138fe78-684d-4f77-9061-a570a9211c82" (UID: "7138fe78-684d-4f77-9061-a570a9211c82"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 06:41:47.492662 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:47.492503 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6pdk9\" (UniqueName: \"kubernetes.io/projected/7138fe78-684d-4f77-9061-a570a9211c82-kube-api-access-6pdk9\") pod \"7138fe78-684d-4f77-9061-a570a9211c82\" (UID: \"7138fe78-684d-4f77-9061-a570a9211c82\") " Sep 18 06:41:47.493093 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:47.493071 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7138fe78-684d-4f77-9061-a570a9211c82-plugins-config-volume\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:41:47.496043 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:47.495766 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7138fe78-684d-4f77-9061-a570a9211c82-kube-api-access-6pdk9" (OuterVolumeSpecName: "kube-api-access-6pdk9") pod "7138fe78-684d-4f77-9061-a570a9211c82" (UID: "7138fe78-684d-4f77-9061-a570a9211c82"). InnerVolumeSpecName "kube-api-access-6pdk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 06:41:47.593735 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:47.593703 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pdk9\" (UniqueName: \"kubernetes.io/projected/7138fe78-684d-4f77-9061-a570a9211c82-kube-api-access-6pdk9\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:41:47.887698 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:47.887658 2567 generic.go:356] "Generic (PLEG): container finished" podID="7138fe78-684d-4f77-9061-a570a9211c82" containerID="ad08d9df5f0f35ba5a678b90859b58d01b6350d729203cddfbe249d33e1a3b1e" exitCode=0 Sep 18 06:41:47.887902 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:47.887717 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-994a4fd1-679b5c994-xpcgk" Sep 18 06:41:47.887902 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:47.887745 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-994a4fd1-679b5c994-xpcgk" event={"ID":"7138fe78-684d-4f77-9061-a570a9211c82","Type":"ContainerDied","Data":"ad08d9df5f0f35ba5a678b90859b58d01b6350d729203cddfbe249d33e1a3b1e"} Sep 18 06:41:47.887902 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:47.887813 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-994a4fd1-679b5c994-xpcgk" event={"ID":"7138fe78-684d-4f77-9061-a570a9211c82","Type":"ContainerDied","Data":"dd0e847bc8f7da65e4162c2fcc97fc92c1e017bd9465116a78093bfd85a7d5e5"} Sep 18 06:41:47.887902 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:47.887835 2567 scope.go:117] "RemoveContainer" containerID="ad08d9df5f0f35ba5a678b90859b58d01b6350d729203cddfbe249d33e1a3b1e" Sep 18 06:41:47.897689 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:47.897669 2567 scope.go:117] "RemoveContainer" containerID="ad08d9df5f0f35ba5a678b90859b58d01b6350d729203cddfbe249d33e1a3b1e" Sep 18 06:41:47.897971 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:41:47.897953 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad08d9df5f0f35ba5a678b90859b58d01b6350d729203cddfbe249d33e1a3b1e\": container with ID starting with ad08d9df5f0f35ba5a678b90859b58d01b6350d729203cddfbe249d33e1a3b1e not found: ID does not exist" containerID="ad08d9df5f0f35ba5a678b90859b58d01b6350d729203cddfbe249d33e1a3b1e" Sep 18 06:41:47.898015 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:47.897979 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad08d9df5f0f35ba5a678b90859b58d01b6350d729203cddfbe249d33e1a3b1e"} err="failed to get container status \"ad08d9df5f0f35ba5a678b90859b58d01b6350d729203cddfbe249d33e1a3b1e\": rpc error: code = NotFound desc = could not find container \"ad08d9df5f0f35ba5a678b90859b58d01b6350d729203cddfbe249d33e1a3b1e\": container with ID starting with ad08d9df5f0f35ba5a678b90859b58d01b6350d729203cddfbe249d33e1a3b1e not found: ID does not exist" Sep 18 06:41:47.908871 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:47.908851 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-994a4fd1-679b5c994-xpcgk"] Sep 18 06:41:47.912810 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:47.912788 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-994a4fd1-679b5c994-xpcgk"] Sep 18 06:41:48.084034 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:48.083999 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc"] Sep 18 06:41:48.084943 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:48.084909 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7138fe78-684d-4f77-9061-a570a9211c82" containerName="payload-pre-processing" Sep 18 06:41:48.084943 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:48.084942 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7138fe78-684d-4f77-9061-a570a9211c82" containerName="payload-pre-processing" Sep 18 06:41:48.085115 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:48.084957 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fe6dc00-7ab0-441c-a90c-6a03c6442ead" containerName="revoke-keys" Sep 18 06:41:48.085115 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:48.084967 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fe6dc00-7ab0-441c-a90c-6a03c6442ead" containerName="revoke-keys" Sep 18 06:41:48.085115 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:48.085076 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fe6dc00-7ab0-441c-a90c-6a03c6442ead" containerName="revoke-keys" Sep 18 06:41:48.085115 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:48.085097 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="7138fe78-684d-4f77-9061-a570a9211c82" containerName="payload-pre-processing" Sep 18 06:41:48.091710 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:48.091683 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc" Sep 18 06:41:48.094058 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:48.094030 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-5e4c6960-openshift-default-dockercfg-c2q4z\"" Sep 18 06:41:48.100698 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:48.100556 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc"] Sep 18 06:41:48.199790 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:48.199711 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9aa5f688-009b-4232-8ec7-e007f8f7458d-istio-token\") pod \"e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc\" (UID: \"9aa5f688-009b-4232-8ec7-e007f8f7458d\") " pod="openshift-ingress/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc" Sep 18 06:41:48.199790 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:48.199751 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aa5f688-009b-4232-8ec7-e007f8f7458d-workload-socket\") pod \"e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc\" (UID: \"9aa5f688-009b-4232-8ec7-e007f8f7458d\") " pod="openshift-ingress/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc" Sep 18 06:41:48.199969 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:48.199816 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9aa5f688-009b-4232-8ec7-e007f8f7458d-istio-data\") pod \"e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc\" (UID: \"9aa5f688-009b-4232-8ec7-e007f8f7458d\") " pod="openshift-ingress/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc" Sep 18 06:41:48.199969 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:48.199899 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9aa5f688-009b-4232-8ec7-e007f8f7458d-istiod-ca-cert\") pod \"e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc\" (UID: \"9aa5f688-009b-4232-8ec7-e007f8f7458d\") " pod="openshift-ingress/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc" Sep 18 06:41:48.199969 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:48.199955 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9aa5f688-009b-4232-8ec7-e007f8f7458d-istio-podinfo\") pod \"e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc\" (UID: \"9aa5f688-009b-4232-8ec7-e007f8f7458d\") " pod="openshift-ingress/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc" Sep 18 06:41:48.200069 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:48.199987 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9aa5f688-009b-4232-8ec7-e007f8f7458d-istio-envoy\") pod \"e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc\" (UID: \"9aa5f688-009b-4232-8ec7-e007f8f7458d\") " pod="openshift-ingress/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc" Sep 18 06:41:48.200069 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:48.200017 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aa5f688-009b-4232-8ec7-e007f8f7458d-credential-socket\") pod \"e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc\" (UID: \"9aa5f688-009b-4232-8ec7-e007f8f7458d\") " pod="openshift-ingress/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc" Sep 18 06:41:48.200130 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:48.200069 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9aa5f688-009b-4232-8ec7-e007f8f7458d-workload-certs\") pod \"e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc\" (UID: \"9aa5f688-009b-4232-8ec7-e007f8f7458d\") " pod="openshift-ingress/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc" Sep 18 06:41:48.200130 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:48.200093 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khwt8\" (UniqueName: \"kubernetes.io/projected/9aa5f688-009b-4232-8ec7-e007f8f7458d-kube-api-access-khwt8\") pod \"e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc\" (UID: \"9aa5f688-009b-4232-8ec7-e007f8f7458d\") " pod="openshift-ingress/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc" Sep 18 06:41:48.301338 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:48.301300 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9aa5f688-009b-4232-8ec7-e007f8f7458d-istio-podinfo\") pod \"e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc\" (UID: \"9aa5f688-009b-4232-8ec7-e007f8f7458d\") " pod="openshift-ingress/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc" Sep 18 06:41:48.301338 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:48.301345 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9aa5f688-009b-4232-8ec7-e007f8f7458d-istio-envoy\") pod \"e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc\" (UID: \"9aa5f688-009b-4232-8ec7-e007f8f7458d\") " pod="openshift-ingress/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc" Sep 18 06:41:48.301613 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:48.301389 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aa5f688-009b-4232-8ec7-e007f8f7458d-credential-socket\") pod \"e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc\" (UID: \"9aa5f688-009b-4232-8ec7-e007f8f7458d\") " pod="openshift-ingress/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc" Sep 18 06:41:48.301613 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:48.301423 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9aa5f688-009b-4232-8ec7-e007f8f7458d-workload-certs\") pod \"e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc\" (UID: \"9aa5f688-009b-4232-8ec7-e007f8f7458d\") " pod="openshift-ingress/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc" Sep 18 06:41:48.301613 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:48.301443 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khwt8\" (UniqueName: \"kubernetes.io/projected/9aa5f688-009b-4232-8ec7-e007f8f7458d-kube-api-access-khwt8\") pod \"e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc\" (UID: \"9aa5f688-009b-4232-8ec7-e007f8f7458d\") " pod="openshift-ingress/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc" Sep 18 06:41:48.301613 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:48.301463 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9aa5f688-009b-4232-8ec7-e007f8f7458d-istio-token\") pod \"e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc\" (UID: \"9aa5f688-009b-4232-8ec7-e007f8f7458d\") " pod="openshift-ingress/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc" Sep 18 06:41:48.301613 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:48.301493 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aa5f688-009b-4232-8ec7-e007f8f7458d-workload-socket\") pod \"e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc\" (UID: \"9aa5f688-009b-4232-8ec7-e007f8f7458d\") " pod="openshift-ingress/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc" Sep 18 06:41:48.301613 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:48.301541 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9aa5f688-009b-4232-8ec7-e007f8f7458d-istio-data\") pod \"e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc\" (UID: \"9aa5f688-009b-4232-8ec7-e007f8f7458d\") " pod="openshift-ingress/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc" Sep 18 06:41:48.301613 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:48.301590 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9aa5f688-009b-4232-8ec7-e007f8f7458d-istiod-ca-cert\") pod \"e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc\" (UID: \"9aa5f688-009b-4232-8ec7-e007f8f7458d\") " pod="openshift-ingress/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc" Sep 18 06:41:48.301999 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:48.301939 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aa5f688-009b-4232-8ec7-e007f8f7458d-credential-socket\") pod \"e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc\" (UID: \"9aa5f688-009b-4232-8ec7-e007f8f7458d\") " pod="openshift-ingress/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc" Sep 18 06:41:48.301999 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:48.301969 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9aa5f688-009b-4232-8ec7-e007f8f7458d-workload-certs\") pod \"e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc\" (UID: \"9aa5f688-009b-4232-8ec7-e007f8f7458d\") " pod="openshift-ingress/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc" Sep 18 06:41:48.302112 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:48.302020 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aa5f688-009b-4232-8ec7-e007f8f7458d-workload-socket\") pod \"e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc\" (UID: \"9aa5f688-009b-4232-8ec7-e007f8f7458d\") " pod="openshift-ingress/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc" Sep 18 06:41:48.302165 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:48.302115 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9aa5f688-009b-4232-8ec7-e007f8f7458d-istio-data\") pod \"e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc\" (UID: \"9aa5f688-009b-4232-8ec7-e007f8f7458d\") " pod="openshift-ingress/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc" Sep 18 06:41:48.302268 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:48.302251 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9aa5f688-009b-4232-8ec7-e007f8f7458d-istiod-ca-cert\") pod \"e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc\" (UID: \"9aa5f688-009b-4232-8ec7-e007f8f7458d\") " pod="openshift-ingress/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc" Sep 18 06:41:48.304273 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:48.304248 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9aa5f688-009b-4232-8ec7-e007f8f7458d-istio-envoy\") pod \"e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc\" (UID: \"9aa5f688-009b-4232-8ec7-e007f8f7458d\") " pod="openshift-ingress/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc" Sep 18 06:41:48.304402 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:48.304338 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9aa5f688-009b-4232-8ec7-e007f8f7458d-istio-podinfo\") pod \"e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc\" (UID: \"9aa5f688-009b-4232-8ec7-e007f8f7458d\") " pod="openshift-ingress/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc" Sep 18 06:41:48.309279 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:48.309254 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9aa5f688-009b-4232-8ec7-e007f8f7458d-istio-token\") pod \"e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc\" (UID: \"9aa5f688-009b-4232-8ec7-e007f8f7458d\") " pod="openshift-ingress/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc" Sep 18 06:41:48.309500 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:48.309483 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khwt8\" (UniqueName: \"kubernetes.io/projected/9aa5f688-009b-4232-8ec7-e007f8f7458d-kube-api-access-khwt8\") pod \"e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc\" (UID: \"9aa5f688-009b-4232-8ec7-e007f8f7458d\") " pod="openshift-ingress/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc" Sep 18 06:41:48.405537 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:48.405504 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc" Sep 18 06:41:48.447469 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:48.447436 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fe6dc00-7ab0-441c-a90c-6a03c6442ead" path="/var/lib/kubelet/pods/3fe6dc00-7ab0-441c-a90c-6a03c6442ead/volumes" Sep 18 06:41:48.447888 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:48.447873 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7138fe78-684d-4f77-9061-a570a9211c82" path="/var/lib/kubelet/pods/7138fe78-684d-4f77-9061-a570a9211c82/volumes" Sep 18 06:41:48.684412 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:48.684382 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc"] Sep 18 06:41:48.685172 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:41:48.685128 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aa5f688_009b_4232_8ec7_e007f8f7458d.slice/crio-7cc9cdea328dd4ba4b08a7086bdd83e34135ec3deeaa84d999e95256dd899629 WatchSource:0}: Error finding container 7cc9cdea328dd4ba4b08a7086bdd83e34135ec3deeaa84d999e95256dd899629: Status 404 returned error can't find the container with id 7cc9cdea328dd4ba4b08a7086bdd83e34135ec3deeaa84d999e95256dd899629 Sep 18 06:41:48.687861 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:48.687823 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:41:48.687939 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:48.687907 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:41:48.687999 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:48.687954 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:41:48.896857 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:48.896815 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc" event={"ID":"9aa5f688-009b-4232-8ec7-e007f8f7458d","Type":"ContainerStarted","Data":"10cdf5bf9e6413f6549226c2b82180e851cf6019c48b9dbbd547e0739245a804"} Sep 18 06:41:48.897053 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:48.896874 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc" event={"ID":"9aa5f688-009b-4232-8ec7-e007f8f7458d","Type":"ContainerStarted","Data":"7cc9cdea328dd4ba4b08a7086bdd83e34135ec3deeaa84d999e95256dd899629"} Sep 18 06:41:48.917657 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:48.917607 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc" podStartSLOduration=0.917591022 podStartE2EDuration="917.591022ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:41:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:41:48.916095238 +0000 UTC m=+1170.655511434" watchObservedRunningTime="2026-09-18 06:41:48.917591022 +0000 UTC m=+1170.657007196" Sep 18 06:41:49.406475 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:49.406432 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc" Sep 18 06:41:49.411685 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:49.411658 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc" Sep 18 06:41:49.903318 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:49.903290 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc" Sep 18 06:41:49.904145 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:49.904128 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc" Sep 18 06:41:52.041993 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:52.041961 2567 patch_prober.go:28] interesting pod/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Sep 18 06:41:52.042456 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:52.042027 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg" podUID="cfc46a52-5b82-4f2a-8b32-3af12d8270b1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 18 06:41:52.637586 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:52.637522 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg" Sep 18 06:41:52.745076 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:52.745042 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfc46a52-5b82-4f2a-8b32-3af12d8270b1-workload-certs\") pod \"cfc46a52-5b82-4f2a-8b32-3af12d8270b1\" (UID: \"cfc46a52-5b82-4f2a-8b32-3af12d8270b1\") " Sep 18 06:41:52.745261 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:52.745102 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q6hpl\" (UniqueName: \"kubernetes.io/projected/cfc46a52-5b82-4f2a-8b32-3af12d8270b1-kube-api-access-q6hpl\") pod \"cfc46a52-5b82-4f2a-8b32-3af12d8270b1\" (UID: \"cfc46a52-5b82-4f2a-8b32-3af12d8270b1\") " Sep 18 06:41:52.745261 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:52.745138 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfc46a52-5b82-4f2a-8b32-3af12d8270b1-istio-envoy\") pod \"cfc46a52-5b82-4f2a-8b32-3af12d8270b1\" (UID: \"cfc46a52-5b82-4f2a-8b32-3af12d8270b1\") " Sep 18 06:41:52.745261 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:52.745183 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfc46a52-5b82-4f2a-8b32-3af12d8270b1-istio-data\") pod \"cfc46a52-5b82-4f2a-8b32-3af12d8270b1\" (UID: \"cfc46a52-5b82-4f2a-8b32-3af12d8270b1\") " Sep 18 06:41:52.745261 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:52.745218 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc46a52-5b82-4f2a-8b32-3af12d8270b1-credential-socket\") pod \"cfc46a52-5b82-4f2a-8b32-3af12d8270b1\" (UID: \"cfc46a52-5b82-4f2a-8b32-3af12d8270b1\") " Sep 18 06:41:52.745537 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:52.745260 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfc46a52-5b82-4f2a-8b32-3af12d8270b1-istio-podinfo\") pod \"cfc46a52-5b82-4f2a-8b32-3af12d8270b1\" (UID: \"cfc46a52-5b82-4f2a-8b32-3af12d8270b1\") " Sep 18 06:41:52.745537 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:52.745338 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfc46a52-5b82-4f2a-8b32-3af12d8270b1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cfc46a52-5b82-4f2a-8b32-3af12d8270b1" (UID: "cfc46a52-5b82-4f2a-8b32-3af12d8270b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:41:52.745537 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:52.745394 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfc46a52-5b82-4f2a-8b32-3af12d8270b1-istiod-ca-cert\") pod \"cfc46a52-5b82-4f2a-8b32-3af12d8270b1\" (UID: \"cfc46a52-5b82-4f2a-8b32-3af12d8270b1\") " Sep 18 06:41:52.745537 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:52.745443 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc46a52-5b82-4f2a-8b32-3af12d8270b1-workload-socket\") pod \"cfc46a52-5b82-4f2a-8b32-3af12d8270b1\" (UID: \"cfc46a52-5b82-4f2a-8b32-3af12d8270b1\") " Sep 18 06:41:52.745537 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:52.745492 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfc46a52-5b82-4f2a-8b32-3af12d8270b1-istio-token\") pod \"cfc46a52-5b82-4f2a-8b32-3af12d8270b1\" (UID: \"cfc46a52-5b82-4f2a-8b32-3af12d8270b1\") " Sep 18 06:41:52.745771 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:52.745595 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfc46a52-5b82-4f2a-8b32-3af12d8270b1-istio-data" (OuterVolumeSpecName: "istio-data") pod "cfc46a52-5b82-4f2a-8b32-3af12d8270b1" (UID: "cfc46a52-5b82-4f2a-8b32-3af12d8270b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:41:52.745771 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:52.745750 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfc46a52-5b82-4f2a-8b32-3af12d8270b1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cfc46a52-5b82-4f2a-8b32-3af12d8270b1" (UID: "cfc46a52-5b82-4f2a-8b32-3af12d8270b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:41:52.745877 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:52.745769 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfc46a52-5b82-4f2a-8b32-3af12d8270b1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cfc46a52-5b82-4f2a-8b32-3af12d8270b1" (UID: "cfc46a52-5b82-4f2a-8b32-3af12d8270b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 06:41:52.745877 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:52.745834 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfc46a52-5b82-4f2a-8b32-3af12d8270b1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cfc46a52-5b82-4f2a-8b32-3af12d8270b1" (UID: "cfc46a52-5b82-4f2a-8b32-3af12d8270b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:41:52.745967 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:52.745953 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfc46a52-5b82-4f2a-8b32-3af12d8270b1-istio-data\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:41:52.746003 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:52.745972 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc46a52-5b82-4f2a-8b32-3af12d8270b1-credential-socket\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:41:52.746003 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:52.745988 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfc46a52-5b82-4f2a-8b32-3af12d8270b1-istiod-ca-cert\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:41:52.746067 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:52.746001 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc46a52-5b82-4f2a-8b32-3af12d8270b1-workload-socket\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:41:52.746067 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:52.746014 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfc46a52-5b82-4f2a-8b32-3af12d8270b1-workload-certs\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:41:52.747472 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:52.747442 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfc46a52-5b82-4f2a-8b32-3af12d8270b1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cfc46a52-5b82-4f2a-8b32-3af12d8270b1" (UID: "cfc46a52-5b82-4f2a-8b32-3af12d8270b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:41:52.747593 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:52.747562 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfc46a52-5b82-4f2a-8b32-3af12d8270b1-kube-api-access-q6hpl" (OuterVolumeSpecName: "kube-api-access-q6hpl") pod "cfc46a52-5b82-4f2a-8b32-3af12d8270b1" (UID: "cfc46a52-5b82-4f2a-8b32-3af12d8270b1"). InnerVolumeSpecName "kube-api-access-q6hpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 06:41:52.747637 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:52.747592 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfc46a52-5b82-4f2a-8b32-3af12d8270b1-istio-token" (OuterVolumeSpecName: "istio-token") pod "cfc46a52-5b82-4f2a-8b32-3af12d8270b1" (UID: "cfc46a52-5b82-4f2a-8b32-3af12d8270b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 06:41:52.747637 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:52.747592 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cfc46a52-5b82-4f2a-8b32-3af12d8270b1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cfc46a52-5b82-4f2a-8b32-3af12d8270b1" (UID: "cfc46a52-5b82-4f2a-8b32-3af12d8270b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 06:41:52.846548 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:52.846512 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfc46a52-5b82-4f2a-8b32-3af12d8270b1-istio-token\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:41:52.846548 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:52.846540 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6hpl\" (UniqueName: \"kubernetes.io/projected/cfc46a52-5b82-4f2a-8b32-3af12d8270b1-kube-api-access-q6hpl\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:41:52.846548 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:52.846549 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfc46a52-5b82-4f2a-8b32-3af12d8270b1-istio-envoy\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:41:52.846766 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:52.846559 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfc46a52-5b82-4f2a-8b32-3af12d8270b1-istio-podinfo\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:41:52.924699 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:52.924620 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5e4c6960-65b7c74fdb-s7485"] Sep 18 06:41:52.925061 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:52.925049 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfc46a52-5b82-4f2a-8b32-3af12d8270b1" containerName="istio-proxy" Sep 18 06:41:52.925108 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:52.925063 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfc46a52-5b82-4f2a-8b32-3af12d8270b1" containerName="istio-proxy" Sep 18 06:41:52.925108 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:52.925077 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfc46a52-5b82-4f2a-8b32-3af12d8270b1" containerName="istio-proxy" Sep 18 06:41:52.925108 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:52.925082 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfc46a52-5b82-4f2a-8b32-3af12d8270b1" containerName="istio-proxy" Sep 18 06:41:52.925199 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:52.925150 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfc46a52-5b82-4f2a-8b32-3af12d8270b1" containerName="istio-proxy" Sep 18 06:41:52.925199 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:52.925159 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfc46a52-5b82-4f2a-8b32-3af12d8270b1" containerName="istio-proxy" Sep 18 06:41:52.928639 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:52.928617 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5e4c6960-65b7c74fdb-s7485" Sep 18 06:41:52.929201 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:52.928917 2567 generic.go:356] "Generic (PLEG): container finished" podID="cfc46a52-5b82-4f2a-8b32-3af12d8270b1" containerID="7fe7cc6416d12461d032af2b77ebac3441a4f359adeb3b320439a03dd16895b1" exitCode=0 Sep 18 06:41:52.929201 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:52.928951 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg" event={"ID":"cfc46a52-5b82-4f2a-8b32-3af12d8270b1","Type":"ContainerDied","Data":"7fe7cc6416d12461d032af2b77ebac3441a4f359adeb3b320439a03dd16895b1"} Sep 18 06:41:52.929201 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:52.928975 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg" Sep 18 06:41:52.929201 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:52.929004 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg" event={"ID":"cfc46a52-5b82-4f2a-8b32-3af12d8270b1","Type":"ContainerDied","Data":"e87fd73c662f97905cdeeb9fada9cf2cdc68473692d59f4029f3e9cec2cc4447"} Sep 18 06:41:52.929201 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:52.929034 2567 scope.go:117] "RemoveContainer" containerID="7fe7cc6416d12461d032af2b77ebac3441a4f359adeb3b320439a03dd16895b1" Sep 18 06:41:52.932001 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:52.931980 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-5e4c6960-dockercfg-hb4w8\"" Sep 18 06:41:52.932111 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:52.932045 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-5e4c6960\"" Sep 18 06:41:52.940881 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:52.940855 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5e4c6960-65b7c74fdb-s7485"] Sep 18 06:41:52.941501 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:52.941481 2567 scope.go:117] "RemoveContainer" containerID="c876d8a975d222bb642ea75a71e4c76c0ee7fe667d53fba0f48749423dd773e4" Sep 18 06:41:52.951831 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:52.951809 2567 scope.go:117] "RemoveContainer" containerID="7fe7cc6416d12461d032af2b77ebac3441a4f359adeb3b320439a03dd16895b1" Sep 18 06:41:52.952130 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:41:52.952112 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fe7cc6416d12461d032af2b77ebac3441a4f359adeb3b320439a03dd16895b1\": container with ID starting with 7fe7cc6416d12461d032af2b77ebac3441a4f359adeb3b320439a03dd16895b1 not found: ID does not exist" containerID="7fe7cc6416d12461d032af2b77ebac3441a4f359adeb3b320439a03dd16895b1" Sep 18 06:41:52.952179 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:52.952140 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fe7cc6416d12461d032af2b77ebac3441a4f359adeb3b320439a03dd16895b1"} err="failed to get container status \"7fe7cc6416d12461d032af2b77ebac3441a4f359adeb3b320439a03dd16895b1\": rpc error: code = NotFound desc = could not find container \"7fe7cc6416d12461d032af2b77ebac3441a4f359adeb3b320439a03dd16895b1\": container with ID starting with 7fe7cc6416d12461d032af2b77ebac3441a4f359adeb3b320439a03dd16895b1 not found: ID does not exist" Sep 18 06:41:52.952179 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:52.952158 2567 scope.go:117] "RemoveContainer" containerID="c876d8a975d222bb642ea75a71e4c76c0ee7fe667d53fba0f48749423dd773e4" Sep 18 06:41:52.952436 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:41:52.952420 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c876d8a975d222bb642ea75a71e4c76c0ee7fe667d53fba0f48749423dd773e4\": container with ID starting with c876d8a975d222bb642ea75a71e4c76c0ee7fe667d53fba0f48749423dd773e4 not found: ID does not exist" containerID="c876d8a975d222bb642ea75a71e4c76c0ee7fe667d53fba0f48749423dd773e4" Sep 18 06:41:52.952501 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:52.952439 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c876d8a975d222bb642ea75a71e4c76c0ee7fe667d53fba0f48749423dd773e4"} err="failed to get container status \"c876d8a975d222bb642ea75a71e4c76c0ee7fe667d53fba0f48749423dd773e4\": rpc error: code = NotFound desc = could not find container \"c876d8a975d222bb642ea75a71e4c76c0ee7fe667d53fba0f48749423dd773e4\": container with ID starting with c876d8a975d222bb642ea75a71e4c76c0ee7fe667d53fba0f48749423dd773e4 not found: ID does not exist" Sep 18 06:41:52.965102 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:52.965079 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg"] Sep 18 06:41:52.973432 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:52.973410 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-994a4fd1-gw-openshift-default-d5785574d-6p4bg"] Sep 18 06:41:53.048415 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:53.048387 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da3bb057-8699-4988-8029-51093187109f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5e4c6960-65b7c74fdb-s7485\" (UID: \"da3bb057-8699-4988-8029-51093187109f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5e4c6960-65b7c74fdb-s7485" Sep 18 06:41:53.048746 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:53.048424 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8426s\" (UniqueName: \"kubernetes.io/projected/da3bb057-8699-4988-8029-51093187109f-kube-api-access-8426s\") pod \"payload-pre-processing-e2e-ait-5e4c6960-65b7c74fdb-s7485\" (UID: \"da3bb057-8699-4988-8029-51093187109f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5e4c6960-65b7c74fdb-s7485" Sep 18 06:41:53.149915 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:53.149882 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da3bb057-8699-4988-8029-51093187109f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5e4c6960-65b7c74fdb-s7485\" (UID: \"da3bb057-8699-4988-8029-51093187109f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5e4c6960-65b7c74fdb-s7485" Sep 18 06:41:53.149915 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:53.149920 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8426s\" (UniqueName: \"kubernetes.io/projected/da3bb057-8699-4988-8029-51093187109f-kube-api-access-8426s\") pod \"payload-pre-processing-e2e-ait-5e4c6960-65b7c74fdb-s7485\" (UID: \"da3bb057-8699-4988-8029-51093187109f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5e4c6960-65b7c74fdb-s7485" Sep 18 06:41:53.150598 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:53.150579 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da3bb057-8699-4988-8029-51093187109f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5e4c6960-65b7c74fdb-s7485\" (UID: \"da3bb057-8699-4988-8029-51093187109f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5e4c6960-65b7c74fdb-s7485" Sep 18 06:41:53.157783 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:53.157757 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8426s\" (UniqueName: \"kubernetes.io/projected/da3bb057-8699-4988-8029-51093187109f-kube-api-access-8426s\") pod \"payload-pre-processing-e2e-ait-5e4c6960-65b7c74fdb-s7485\" (UID: \"da3bb057-8699-4988-8029-51093187109f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5e4c6960-65b7c74fdb-s7485" Sep 18 06:41:53.244199 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:53.244139 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5e4c6960-65b7c74fdb-s7485" Sep 18 06:41:53.519419 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:53.519390 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5e4c6960-65b7c74fdb-s7485"] Sep 18 06:41:53.520327 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:41:53.520304 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda3bb057_8699_4988_8029_51093187109f.slice/crio-2526fb9fc7bc8e8f613c1268726a9adbc431d5bdeecd81a9786e4793c8e69df9 WatchSource:0}: Error finding container 2526fb9fc7bc8e8f613c1268726a9adbc431d5bdeecd81a9786e4793c8e69df9: Status 404 returned error can't find the container with id 2526fb9fc7bc8e8f613c1268726a9adbc431d5bdeecd81a9786e4793c8e69df9 Sep 18 06:41:53.938011 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:53.937977 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5e4c6960-65b7c74fdb-s7485" event={"ID":"da3bb057-8699-4988-8029-51093187109f","Type":"ContainerStarted","Data":"3fcce8d672f5878c28d4ef73bdd7c2c64a2804b54633a5712d33e62dfac1f02a"} Sep 18 06:41:53.938194 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:53.938033 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5e4c6960-65b7c74fdb-s7485" event={"ID":"da3bb057-8699-4988-8029-51093187109f","Type":"ContainerStarted","Data":"2526fb9fc7bc8e8f613c1268726a9adbc431d5bdeecd81a9786e4793c8e69df9"} Sep 18 06:41:53.938194 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:53.938072 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5e4c6960-65b7c74fdb-s7485" Sep 18 06:41:53.957704 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:53.957641 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-5e4c6960-65b7c74fdb-s7485" podStartSLOduration=1.957617594 podStartE2EDuration="1.957617594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:41:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:41:53.955743294 +0000 UTC m=+1175.695159485" watchObservedRunningTime="2026-09-18 06:41:53.957617594 +0000 UTC m=+1175.697033772" Sep 18 06:41:54.446836 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:41:54.446801 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfc46a52-5b82-4f2a-8b32-3af12d8270b1" path="/var/lib/kubelet/pods/cfc46a52-5b82-4f2a-8b32-3af12d8270b1/volumes" Sep 18 06:42:04.946007 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:04.945971 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5e4c6960-65b7c74fdb-s7485" Sep 18 06:42:13.062036 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:13.061999 2567 generic.go:356] "Generic (PLEG): container finished" podID="9aa5f688-009b-4232-8ec7-e007f8f7458d" containerID="10cdf5bf9e6413f6549226c2b82180e851cf6019c48b9dbbd547e0739245a804" exitCode=0 Sep 18 06:42:13.062451 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:13.062072 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc" event={"ID":"9aa5f688-009b-4232-8ec7-e007f8f7458d","Type":"ContainerDied","Data":"10cdf5bf9e6413f6549226c2b82180e851cf6019c48b9dbbd547e0739245a804"} Sep 18 06:42:13.062501 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:13.062473 2567 scope.go:117] "RemoveContainer" containerID="10cdf5bf9e6413f6549226c2b82180e851cf6019c48b9dbbd547e0739245a804" Sep 18 06:42:13.063657 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:13.063631 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:42:13.063721 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:13.063694 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:42:13.063758 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:13.063722 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:42:14.076264 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:14.075254 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc" event={"ID":"9aa5f688-009b-4232-8ec7-e007f8f7458d","Type":"ContainerStarted","Data":"f7ba0012e72fb97e77e1e784c962684f9ec176eedc77632c0d3dc7fed6776b9a"} Sep 18 06:42:14.406438 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:14.406397 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc" Sep 18 06:42:14.406699 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:14.406673 2567 patch_prober.go:28] interesting pod/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Sep 18 06:42:14.406803 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:14.406743 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc" podUID="9aa5f688-009b-4232-8ec7-e007f8f7458d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Sep 18 06:42:15.406032 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:15.405997 2567 patch_prober.go:28] interesting pod/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Sep 18 06:42:15.406433 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:15.406059 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc" podUID="9aa5f688-009b-4232-8ec7-e007f8f7458d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Sep 18 06:42:16.406706 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:16.406673 2567 patch_prober.go:28] interesting pod/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Sep 18 06:42:16.407071 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:16.406728 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc" podUID="9aa5f688-009b-4232-8ec7-e007f8f7458d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Sep 18 06:42:17.406327 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:17.406290 2567 patch_prober.go:28] interesting pod/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Sep 18 06:42:17.406528 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:17.406350 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc" podUID="9aa5f688-009b-4232-8ec7-e007f8f7458d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Sep 18 06:42:18.406463 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:18.406425 2567 patch_prober.go:28] interesting pod/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Sep 18 06:42:18.406837 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:18.406488 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc" podUID="9aa5f688-009b-4232-8ec7-e007f8f7458d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Sep 18 06:42:19.462267 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:19.462239 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc" Sep 18 06:42:19.462728 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:19.462706 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc" Sep 18 06:42:19.464904 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:19.464882 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc" Sep 18 06:42:20.453177 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:20.453146 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjr4h_eb059951-e263-40da-9155-4fd0cf242820/ovn-acl-logging/0.log" Sep 18 06:42:20.456239 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:20.456216 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjr4h_eb059951-e263-40da-9155-4fd0cf242820/ovn-acl-logging/0.log" Sep 18 06:42:20.995509 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:20.995474 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5e4c6960-fsgrk"] Sep 18 06:42:20.998832 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:20.998809 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5e4c6960-fsgrk" Sep 18 06:42:21.001200 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:21.001176 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 18 06:42:21.001311 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:21.001233 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-88t7h\"" Sep 18 06:42:21.006753 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:21.006731 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5e4c6960-fsgrk"] Sep 18 06:42:21.020327 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:21.020284 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91ca6655-1c9e-4ecb-8efb-7bfb85ee962d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5e4c6960-fsgrk\" (UID: \"91ca6655-1c9e-4ecb-8efb-7bfb85ee962d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5e4c6960-fsgrk" Sep 18 06:42:21.120820 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:21.120792 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91ca6655-1c9e-4ecb-8efb-7bfb85ee962d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5e4c6960-fsgrk\" (UID: \"91ca6655-1c9e-4ecb-8efb-7bfb85ee962d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5e4c6960-fsgrk" Sep 18 06:42:21.121425 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:21.121405 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91ca6655-1c9e-4ecb-8efb-7bfb85ee962d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5e4c6960-fsgrk\" (UID: \"91ca6655-1c9e-4ecb-8efb-7bfb85ee962d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5e4c6960-fsgrk" Sep 18 06:42:21.313553 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:21.313461 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5e4c6960-fsgrk" Sep 18 06:42:21.586729 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:21.586699 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5e4c6960-fsgrk"] Sep 18 06:42:21.588741 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:42:21.588717 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91ca6655_1c9e_4ecb_8efb_7bfb85ee962d.slice/crio-e98181ec463781bd8a4012511649eff4b53834f699bd5387e7387c7e95098aaf WatchSource:0}: Error finding container e98181ec463781bd8a4012511649eff4b53834f699bd5387e7387c7e95098aaf: Status 404 returned error can't find the container with id e98181ec463781bd8a4012511649eff4b53834f699bd5387e7387c7e95098aaf Sep 18 06:42:22.124579 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:22.124480 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5e4c6960-fsgrk" event={"ID":"91ca6655-1c9e-4ecb-8efb-7bfb85ee962d","Type":"ContainerStarted","Data":"1777d7c4b9e38d06ee2d378bfeabe1b3fd2413d4e56126161620d6eef52d5db7"} Sep 18 06:42:22.124579 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:22.124534 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5e4c6960-fsgrk" event={"ID":"91ca6655-1c9e-4ecb-8efb-7bfb85ee962d","Type":"ContainerStarted","Data":"e98181ec463781bd8a4012511649eff4b53834f699bd5387e7387c7e95098aaf"} Sep 18 06:42:22.140907 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:22.140860 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5e4c6960-fsgrk" podStartSLOduration=2.14084518 podStartE2EDuration="2.14084518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:42:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:42:22.1376529 +0000 UTC m=+1203.877069099" watchObservedRunningTime="2026-09-18 06:42:22.14084518 +0000 UTC m=+1203.880261355" Sep 18 06:42:23.131584 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:23.131552 2567 generic.go:356] "Generic (PLEG): container finished" podID="91ca6655-1c9e-4ecb-8efb-7bfb85ee962d" containerID="1777d7c4b9e38d06ee2d378bfeabe1b3fd2413d4e56126161620d6eef52d5db7" exitCode=0 Sep 18 06:42:23.131968 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:23.131596 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5e4c6960-fsgrk" event={"ID":"91ca6655-1c9e-4ecb-8efb-7bfb85ee962d","Type":"ContainerDied","Data":"1777d7c4b9e38d06ee2d378bfeabe1b3fd2413d4e56126161620d6eef52d5db7"} Sep 18 06:42:24.270576 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:24.270552 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5e4c6960-fsgrk" Sep 18 06:42:24.357495 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:24.357457 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91ca6655-1c9e-4ecb-8efb-7bfb85ee962d-maas-api-service-ca\") pod \"91ca6655-1c9e-4ecb-8efb-7bfb85ee962d\" (UID: \"91ca6655-1c9e-4ecb-8efb-7bfb85ee962d\") " Sep 18 06:42:24.357821 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:24.357799 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91ca6655-1c9e-4ecb-8efb-7bfb85ee962d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "91ca6655-1c9e-4ecb-8efb-7bfb85ee962d" (UID: "91ca6655-1c9e-4ecb-8efb-7bfb85ee962d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 06:42:24.459012 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:24.458987 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91ca6655-1c9e-4ecb-8efb-7bfb85ee962d-maas-api-service-ca\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:42:25.144475 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:25.144444 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5e4c6960-fsgrk" Sep 18 06:42:25.144646 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:25.144470 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5e4c6960-fsgrk" event={"ID":"91ca6655-1c9e-4ecb-8efb-7bfb85ee962d","Type":"ContainerDied","Data":"e98181ec463781bd8a4012511649eff4b53834f699bd5387e7387c7e95098aaf"} Sep 18 06:42:25.144646 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:25.144516 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e98181ec463781bd8a4012511649eff4b53834f699bd5387e7387c7e95098aaf" Sep 18 06:42:31.987730 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:31.987693 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5e4c6960-65b7c74fdb-s7485"] Sep 18 06:42:31.988235 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:31.987979 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-5e4c6960-65b7c74fdb-s7485" podUID="da3bb057-8699-4988-8029-51093187109f" containerName="payload-pre-processing" containerID="cri-o://3fcce8d672f5878c28d4ef73bdd7c2c64a2804b54633a5712d33e62dfac1f02a" gracePeriod=30 Sep 18 06:42:32.120507 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:32.120472 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5e4c6960-fsgrk"] Sep 18 06:42:32.123038 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:32.123015 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5e4c6960-fsgrk"] Sep 18 06:42:32.188951 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:32.188919 2567 generic.go:356] "Generic (PLEG): container finished" podID="da3bb057-8699-4988-8029-51093187109f" containerID="3fcce8d672f5878c28d4ef73bdd7c2c64a2804b54633a5712d33e62dfac1f02a" exitCode=0 Sep 18 06:42:32.189117 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:32.188985 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5e4c6960-65b7c74fdb-s7485" event={"ID":"da3bb057-8699-4988-8029-51093187109f","Type":"ContainerDied","Data":"3fcce8d672f5878c28d4ef73bdd7c2c64a2804b54633a5712d33e62dfac1f02a"} Sep 18 06:42:32.259835 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:32.259810 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5e4c6960-65b7c74fdb-s7485" Sep 18 06:42:32.336911 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:32.336881 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da3bb057-8699-4988-8029-51093187109f-plugins-config-volume\") pod \"da3bb057-8699-4988-8029-51093187109f\" (UID: \"da3bb057-8699-4988-8029-51093187109f\") " Sep 18 06:42:32.337122 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:32.336930 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8426s\" (UniqueName: \"kubernetes.io/projected/da3bb057-8699-4988-8029-51093187109f-kube-api-access-8426s\") pod \"da3bb057-8699-4988-8029-51093187109f\" (UID: \"da3bb057-8699-4988-8029-51093187109f\") " Sep 18 06:42:32.337302 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:32.337277 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da3bb057-8699-4988-8029-51093187109f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "da3bb057-8699-4988-8029-51093187109f" (UID: "da3bb057-8699-4988-8029-51093187109f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 06:42:32.339335 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:32.339309 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da3bb057-8699-4988-8029-51093187109f-kube-api-access-8426s" (OuterVolumeSpecName: "kube-api-access-8426s") pod "da3bb057-8699-4988-8029-51093187109f" (UID: "da3bb057-8699-4988-8029-51093187109f"). InnerVolumeSpecName "kube-api-access-8426s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 06:42:32.438055 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:32.438024 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8426s\" (UniqueName: \"kubernetes.io/projected/da3bb057-8699-4988-8029-51093187109f-kube-api-access-8426s\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:42:32.438055 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:32.438049 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da3bb057-8699-4988-8029-51093187109f-plugins-config-volume\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:42:32.446312 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:32.446284 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91ca6655-1c9e-4ecb-8efb-7bfb85ee962d" path="/var/lib/kubelet/pods/91ca6655-1c9e-4ecb-8efb-7bfb85ee962d/volumes" Sep 18 06:42:33.194274 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:33.194239 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc"] Sep 18 06:42:33.194768 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:33.194514 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc" podUID="9aa5f688-009b-4232-8ec7-e007f8f7458d" containerName="istio-proxy" containerID="cri-o://f7ba0012e72fb97e77e1e784c962684f9ec176eedc77632c0d3dc7fed6776b9a" gracePeriod=30 Sep 18 06:42:33.198956 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:33.198934 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5e4c6960-65b7c74fdb-s7485" Sep 18 06:42:33.199082 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:33.198963 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5e4c6960-65b7c74fdb-s7485" event={"ID":"da3bb057-8699-4988-8029-51093187109f","Type":"ContainerDied","Data":"2526fb9fc7bc8e8f613c1268726a9adbc431d5bdeecd81a9786e4793c8e69df9"} Sep 18 06:42:33.199082 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:33.199046 2567 scope.go:117] "RemoveContainer" containerID="3fcce8d672f5878c28d4ef73bdd7c2c64a2804b54633a5712d33e62dfac1f02a" Sep 18 06:42:33.226573 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:33.226544 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5e4c6960-65b7c74fdb-s7485"] Sep 18 06:42:33.230779 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:33.230752 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5e4c6960-65b7c74fdb-s7485"] Sep 18 06:42:33.884764 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:33.884728 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw"] Sep 18 06:42:33.885320 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:33.885300 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91ca6655-1c9e-4ecb-8efb-7bfb85ee962d" containerName="revoke-keys" Sep 18 06:42:33.885320 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:33.885320 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="91ca6655-1c9e-4ecb-8efb-7bfb85ee962d" containerName="revoke-keys" Sep 18 06:42:33.885543 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:33.885330 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da3bb057-8699-4988-8029-51093187109f" containerName="payload-pre-processing" Sep 18 06:42:33.885543 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:33.885336 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="da3bb057-8699-4988-8029-51093187109f" containerName="payload-pre-processing" Sep 18 06:42:33.885543 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:33.885447 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="91ca6655-1c9e-4ecb-8efb-7bfb85ee962d" containerName="revoke-keys" Sep 18 06:42:33.885543 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:33.885460 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="da3bb057-8699-4988-8029-51093187109f" containerName="payload-pre-processing" Sep 18 06:42:33.888545 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:33.888521 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw" Sep 18 06:42:33.891020 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:33.890997 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-246af4f6-openshift-default-dockercfg-b59kz\"" Sep 18 06:42:33.901821 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:33.901798 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw"] Sep 18 06:42:33.953632 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:33.953602 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f1cb84-a6be-4c3d-bc42-0876725c58aa-workload-socket\") pod \"e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw\" (UID: \"67f1cb84-a6be-4c3d-bc42-0876725c58aa\") " pod="openshift-ingress/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw" Sep 18 06:42:33.953813 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:33.953637 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67f1cb84-a6be-4c3d-bc42-0876725c58aa-workload-certs\") pod \"e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw\" (UID: \"67f1cb84-a6be-4c3d-bc42-0876725c58aa\") " pod="openshift-ingress/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw" Sep 18 06:42:33.953813 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:33.953683 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pl4v\" (UniqueName: \"kubernetes.io/projected/67f1cb84-a6be-4c3d-bc42-0876725c58aa-kube-api-access-6pl4v\") pod \"e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw\" (UID: \"67f1cb84-a6be-4c3d-bc42-0876725c58aa\") " pod="openshift-ingress/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw" Sep 18 06:42:33.953813 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:33.953779 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67f1cb84-a6be-4c3d-bc42-0876725c58aa-istio-podinfo\") pod \"e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw\" (UID: \"67f1cb84-a6be-4c3d-bc42-0876725c58aa\") " pod="openshift-ingress/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw" Sep 18 06:42:33.953986 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:33.953842 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67f1cb84-a6be-4c3d-bc42-0876725c58aa-istio-token\") pod \"e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw\" (UID: \"67f1cb84-a6be-4c3d-bc42-0876725c58aa\") " pod="openshift-ingress/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw" Sep 18 06:42:33.953986 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:33.953883 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67f1cb84-a6be-4c3d-bc42-0876725c58aa-istio-data\") pod \"e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw\" (UID: \"67f1cb84-a6be-4c3d-bc42-0876725c58aa\") " pod="openshift-ingress/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw" Sep 18 06:42:33.953986 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:33.953915 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f1cb84-a6be-4c3d-bc42-0876725c58aa-credential-socket\") pod \"e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw\" (UID: \"67f1cb84-a6be-4c3d-bc42-0876725c58aa\") " pod="openshift-ingress/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw" Sep 18 06:42:33.953986 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:33.953950 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67f1cb84-a6be-4c3d-bc42-0876725c58aa-istiod-ca-cert\") pod \"e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw\" (UID: \"67f1cb84-a6be-4c3d-bc42-0876725c58aa\") " pod="openshift-ingress/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw" Sep 18 06:42:33.953986 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:33.953978 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67f1cb84-a6be-4c3d-bc42-0876725c58aa-istio-envoy\") pod \"e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw\" (UID: \"67f1cb84-a6be-4c3d-bc42-0876725c58aa\") " pod="openshift-ingress/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw" Sep 18 06:42:34.055264 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:34.055218 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67f1cb84-a6be-4c3d-bc42-0876725c58aa-istiod-ca-cert\") pod \"e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw\" (UID: \"67f1cb84-a6be-4c3d-bc42-0876725c58aa\") " pod="openshift-ingress/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw" Sep 18 06:42:34.055476 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:34.055286 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67f1cb84-a6be-4c3d-bc42-0876725c58aa-istio-envoy\") pod \"e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw\" (UID: \"67f1cb84-a6be-4c3d-bc42-0876725c58aa\") " pod="openshift-ingress/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw" Sep 18 06:42:34.055476 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:34.055320 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f1cb84-a6be-4c3d-bc42-0876725c58aa-workload-socket\") pod \"e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw\" (UID: \"67f1cb84-a6be-4c3d-bc42-0876725c58aa\") " pod="openshift-ingress/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw" Sep 18 06:42:34.055476 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:34.055350 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67f1cb84-a6be-4c3d-bc42-0876725c58aa-workload-certs\") pod \"e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw\" (UID: \"67f1cb84-a6be-4c3d-bc42-0876725c58aa\") " pod="openshift-ingress/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw" Sep 18 06:42:34.055476 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:34.055431 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pl4v\" (UniqueName: \"kubernetes.io/projected/67f1cb84-a6be-4c3d-bc42-0876725c58aa-kube-api-access-6pl4v\") pod \"e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw\" (UID: \"67f1cb84-a6be-4c3d-bc42-0876725c58aa\") " pod="openshift-ingress/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw" Sep 18 06:42:34.055705 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:34.055499 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67f1cb84-a6be-4c3d-bc42-0876725c58aa-istio-podinfo\") pod \"e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw\" (UID: \"67f1cb84-a6be-4c3d-bc42-0876725c58aa\") " pod="openshift-ingress/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw" Sep 18 06:42:34.055705 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:34.055561 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67f1cb84-a6be-4c3d-bc42-0876725c58aa-istio-token\") pod \"e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw\" (UID: \"67f1cb84-a6be-4c3d-bc42-0876725c58aa\") " pod="openshift-ingress/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw" Sep 18 06:42:34.055705 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:34.055608 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67f1cb84-a6be-4c3d-bc42-0876725c58aa-istio-data\") pod \"e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw\" (UID: \"67f1cb84-a6be-4c3d-bc42-0876725c58aa\") " pod="openshift-ingress/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw" Sep 18 06:42:34.055705 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:34.055659 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f1cb84-a6be-4c3d-bc42-0876725c58aa-credential-socket\") pod \"e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw\" (UID: \"67f1cb84-a6be-4c3d-bc42-0876725c58aa\") " pod="openshift-ingress/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw" Sep 18 06:42:34.056144 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:34.056119 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67f1cb84-a6be-4c3d-bc42-0876725c58aa-istiod-ca-cert\") pod \"e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw\" (UID: \"67f1cb84-a6be-4c3d-bc42-0876725c58aa\") " pod="openshift-ingress/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw" Sep 18 06:42:34.056214 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:34.056150 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f1cb84-a6be-4c3d-bc42-0876725c58aa-credential-socket\") pod \"e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw\" (UID: \"67f1cb84-a6be-4c3d-bc42-0876725c58aa\") " pod="openshift-ingress/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw" Sep 18 06:42:34.057004 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:34.056972 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67f1cb84-a6be-4c3d-bc42-0876725c58aa-istio-data\") pod \"e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw\" (UID: \"67f1cb84-a6be-4c3d-bc42-0876725c58aa\") " pod="openshift-ingress/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw" Sep 18 06:42:34.057302 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:34.057277 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67f1cb84-a6be-4c3d-bc42-0876725c58aa-workload-certs\") pod \"e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw\" (UID: \"67f1cb84-a6be-4c3d-bc42-0876725c58aa\") " pod="openshift-ingress/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw" Sep 18 06:42:34.057435 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:34.057299 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f1cb84-a6be-4c3d-bc42-0876725c58aa-workload-socket\") pod \"e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw\" (UID: \"67f1cb84-a6be-4c3d-bc42-0876725c58aa\") " pod="openshift-ingress/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw" Sep 18 06:42:34.059713 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:34.059690 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67f1cb84-a6be-4c3d-bc42-0876725c58aa-istio-envoy\") pod \"e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw\" (UID: \"67f1cb84-a6be-4c3d-bc42-0876725c58aa\") " pod="openshift-ingress/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw" Sep 18 06:42:34.059922 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:34.059904 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67f1cb84-a6be-4c3d-bc42-0876725c58aa-istio-podinfo\") pod \"e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw\" (UID: \"67f1cb84-a6be-4c3d-bc42-0876725c58aa\") " pod="openshift-ingress/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw" Sep 18 06:42:34.063906 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:34.063877 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67f1cb84-a6be-4c3d-bc42-0876725c58aa-istio-token\") pod \"e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw\" (UID: \"67f1cb84-a6be-4c3d-bc42-0876725c58aa\") " pod="openshift-ingress/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw" Sep 18 06:42:34.063986 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:34.063904 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pl4v\" (UniqueName: \"kubernetes.io/projected/67f1cb84-a6be-4c3d-bc42-0876725c58aa-kube-api-access-6pl4v\") pod \"e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw\" (UID: \"67f1cb84-a6be-4c3d-bc42-0876725c58aa\") " pod="openshift-ingress/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw" Sep 18 06:42:34.200685 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:34.200593 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw" Sep 18 06:42:34.446516 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:34.446478 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da3bb057-8699-4988-8029-51093187109f" path="/var/lib/kubelet/pods/da3bb057-8699-4988-8029-51093187109f/volumes" Sep 18 06:42:34.465262 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:34.465166 2567 patch_prober.go:28] interesting pod/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 18 06:42:34.465262 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:34.465217 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc" podUID="9aa5f688-009b-4232-8ec7-e007f8f7458d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 18 06:42:34.477993 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:34.477971 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw"] Sep 18 06:42:34.479572 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:42:34.479542 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67f1cb84_a6be_4c3d_bc42_0876725c58aa.slice/crio-58f46f9f19ee123b3616ebda5ae8e752a8ed33cb6e5fd1d79fd2c7a5a664b353 WatchSource:0}: Error finding container 58f46f9f19ee123b3616ebda5ae8e752a8ed33cb6e5fd1d79fd2c7a5a664b353: Status 404 returned error can't find the container with id 58f46f9f19ee123b3616ebda5ae8e752a8ed33cb6e5fd1d79fd2c7a5a664b353 Sep 18 06:42:34.482226 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:34.482189 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:42:34.482308 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:34.482279 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:42:34.482351 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:34.482321 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:42:35.213986 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:35.213948 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw" event={"ID":"67f1cb84-a6be-4c3d-bc42-0876725c58aa","Type":"ContainerStarted","Data":"5fb8e3c0deff4238e75f9b9584d25e506613feb4210d47c52ad032faf4c9260a"} Sep 18 06:42:35.214400 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:35.214030 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw" event={"ID":"67f1cb84-a6be-4c3d-bc42-0876725c58aa","Type":"ContainerStarted","Data":"58f46f9f19ee123b3616ebda5ae8e752a8ed33cb6e5fd1d79fd2c7a5a664b353"} Sep 18 06:42:35.233310 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:35.233265 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw" podStartSLOduration=2.233252249 podStartE2EDuration="2.233252249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:42:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:42:35.232038508 +0000 UTC m=+1216.971454706" watchObservedRunningTime="2026-09-18 06:42:35.233252249 +0000 UTC m=+1216.972668423" Sep 18 06:42:36.201135 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:36.201101 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw" Sep 18 06:42:36.206507 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:36.206485 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw" Sep 18 06:42:36.220073 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:36.220047 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw" Sep 18 06:42:36.220964 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:36.220946 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw" Sep 18 06:42:38.453152 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:38.453126 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc" Sep 18 06:42:38.611947 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:38.611912 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9aa5f688-009b-4232-8ec7-e007f8f7458d-istio-envoy\") pod \"9aa5f688-009b-4232-8ec7-e007f8f7458d\" (UID: \"9aa5f688-009b-4232-8ec7-e007f8f7458d\") " Sep 18 06:42:38.612107 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:38.611961 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9aa5f688-009b-4232-8ec7-e007f8f7458d-workload-certs\") pod \"9aa5f688-009b-4232-8ec7-e007f8f7458d\" (UID: \"9aa5f688-009b-4232-8ec7-e007f8f7458d\") " Sep 18 06:42:38.612107 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:38.612007 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aa5f688-009b-4232-8ec7-e007f8f7458d-workload-socket\") pod \"9aa5f688-009b-4232-8ec7-e007f8f7458d\" (UID: \"9aa5f688-009b-4232-8ec7-e007f8f7458d\") " Sep 18 06:42:38.612107 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:38.612027 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-khwt8\" (UniqueName: \"kubernetes.io/projected/9aa5f688-009b-4232-8ec7-e007f8f7458d-kube-api-access-khwt8\") pod \"9aa5f688-009b-4232-8ec7-e007f8f7458d\" (UID: \"9aa5f688-009b-4232-8ec7-e007f8f7458d\") " Sep 18 06:42:38.612107 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:38.612090 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aa5f688-009b-4232-8ec7-e007f8f7458d-credential-socket\") pod \"9aa5f688-009b-4232-8ec7-e007f8f7458d\" (UID: \"9aa5f688-009b-4232-8ec7-e007f8f7458d\") " Sep 18 06:42:38.612335 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:38.612139 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9aa5f688-009b-4232-8ec7-e007f8f7458d-istiod-ca-cert\") pod \"9aa5f688-009b-4232-8ec7-e007f8f7458d\" (UID: \"9aa5f688-009b-4232-8ec7-e007f8f7458d\") " Sep 18 06:42:38.612335 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:38.612162 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9aa5f688-009b-4232-8ec7-e007f8f7458d-istio-token\") pod \"9aa5f688-009b-4232-8ec7-e007f8f7458d\" (UID: \"9aa5f688-009b-4232-8ec7-e007f8f7458d\") " Sep 18 06:42:38.612335 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:38.612231 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9aa5f688-009b-4232-8ec7-e007f8f7458d-istio-data\") pod \"9aa5f688-009b-4232-8ec7-e007f8f7458d\" (UID: \"9aa5f688-009b-4232-8ec7-e007f8f7458d\") " Sep 18 06:42:38.612335 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:38.612256 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9aa5f688-009b-4232-8ec7-e007f8f7458d-istio-podinfo\") pod \"9aa5f688-009b-4232-8ec7-e007f8f7458d\" (UID: \"9aa5f688-009b-4232-8ec7-e007f8f7458d\") " Sep 18 06:42:38.612335 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:38.612268 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aa5f688-009b-4232-8ec7-e007f8f7458d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9aa5f688-009b-4232-8ec7-e007f8f7458d" (UID: "9aa5f688-009b-4232-8ec7-e007f8f7458d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:42:38.612335 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:38.612280 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aa5f688-009b-4232-8ec7-e007f8f7458d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9aa5f688-009b-4232-8ec7-e007f8f7458d" (UID: "9aa5f688-009b-4232-8ec7-e007f8f7458d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:42:38.612665 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:38.612435 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aa5f688-009b-4232-8ec7-e007f8f7458d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9aa5f688-009b-4232-8ec7-e007f8f7458d" (UID: "9aa5f688-009b-4232-8ec7-e007f8f7458d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:42:38.612665 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:38.612528 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aa5f688-009b-4232-8ec7-e007f8f7458d-istio-data" (OuterVolumeSpecName: "istio-data") pod "9aa5f688-009b-4232-8ec7-e007f8f7458d" (UID: "9aa5f688-009b-4232-8ec7-e007f8f7458d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:42:38.612665 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:38.612540 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9aa5f688-009b-4232-8ec7-e007f8f7458d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9aa5f688-009b-4232-8ec7-e007f8f7458d" (UID: "9aa5f688-009b-4232-8ec7-e007f8f7458d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 06:42:38.612818 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:38.612706 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9aa5f688-009b-4232-8ec7-e007f8f7458d-istio-data\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:42:38.612818 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:38.612727 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9aa5f688-009b-4232-8ec7-e007f8f7458d-workload-certs\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:42:38.612818 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:38.612742 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aa5f688-009b-4232-8ec7-e007f8f7458d-workload-socket\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:42:38.612818 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:38.612756 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aa5f688-009b-4232-8ec7-e007f8f7458d-credential-socket\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:42:38.612818 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:38.612770 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9aa5f688-009b-4232-8ec7-e007f8f7458d-istiod-ca-cert\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:42:38.614317 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:38.614286 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aa5f688-009b-4232-8ec7-e007f8f7458d-istio-token" (OuterVolumeSpecName: "istio-token") pod "9aa5f688-009b-4232-8ec7-e007f8f7458d" (UID: "9aa5f688-009b-4232-8ec7-e007f8f7458d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 06:42:38.614473 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:38.614427 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aa5f688-009b-4232-8ec7-e007f8f7458d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9aa5f688-009b-4232-8ec7-e007f8f7458d" (UID: "9aa5f688-009b-4232-8ec7-e007f8f7458d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:42:38.614473 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:38.614446 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9aa5f688-009b-4232-8ec7-e007f8f7458d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9aa5f688-009b-4232-8ec7-e007f8f7458d" (UID: "9aa5f688-009b-4232-8ec7-e007f8f7458d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 06:42:38.614567 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:38.614484 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aa5f688-009b-4232-8ec7-e007f8f7458d-kube-api-access-khwt8" (OuterVolumeSpecName: "kube-api-access-khwt8") pod "9aa5f688-009b-4232-8ec7-e007f8f7458d" (UID: "9aa5f688-009b-4232-8ec7-e007f8f7458d"). InnerVolumeSpecName "kube-api-access-khwt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 06:42:38.713800 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:38.713764 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9aa5f688-009b-4232-8ec7-e007f8f7458d-istio-token\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:42:38.713800 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:38.713794 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9aa5f688-009b-4232-8ec7-e007f8f7458d-istio-podinfo\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:42:38.713800 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:38.713805 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9aa5f688-009b-4232-8ec7-e007f8f7458d-istio-envoy\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:42:38.714036 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:38.713813 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khwt8\" (UniqueName: \"kubernetes.io/projected/9aa5f688-009b-4232-8ec7-e007f8f7458d-kube-api-access-khwt8\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:42:39.243040 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:39.243004 2567 generic.go:356] "Generic (PLEG): container finished" podID="9aa5f688-009b-4232-8ec7-e007f8f7458d" containerID="f7ba0012e72fb97e77e1e784c962684f9ec176eedc77632c0d3dc7fed6776b9a" exitCode=0 Sep 18 06:42:39.243228 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:39.243058 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc" Sep 18 06:42:39.243228 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:39.243088 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc" event={"ID":"9aa5f688-009b-4232-8ec7-e007f8f7458d","Type":"ContainerDied","Data":"f7ba0012e72fb97e77e1e784c962684f9ec176eedc77632c0d3dc7fed6776b9a"} Sep 18 06:42:39.243228 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:39.243149 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc" event={"ID":"9aa5f688-009b-4232-8ec7-e007f8f7458d","Type":"ContainerDied","Data":"7cc9cdea328dd4ba4b08a7086bdd83e34135ec3deeaa84d999e95256dd899629"} Sep 18 06:42:39.243228 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:39.243181 2567 scope.go:117] "RemoveContainer" containerID="f7ba0012e72fb97e77e1e784c962684f9ec176eedc77632c0d3dc7fed6776b9a" Sep 18 06:42:39.253537 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:39.253514 2567 scope.go:117] "RemoveContainer" containerID="10cdf5bf9e6413f6549226c2b82180e851cf6019c48b9dbbd547e0739245a804" Sep 18 06:42:39.262222 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:39.262206 2567 scope.go:117] "RemoveContainer" containerID="f7ba0012e72fb97e77e1e784c962684f9ec176eedc77632c0d3dc7fed6776b9a" Sep 18 06:42:39.262545 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:42:39.262527 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7ba0012e72fb97e77e1e784c962684f9ec176eedc77632c0d3dc7fed6776b9a\": container with ID starting with f7ba0012e72fb97e77e1e784c962684f9ec176eedc77632c0d3dc7fed6776b9a not found: ID does not exist" containerID="f7ba0012e72fb97e77e1e784c962684f9ec176eedc77632c0d3dc7fed6776b9a" Sep 18 06:42:39.262626 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:39.262557 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7ba0012e72fb97e77e1e784c962684f9ec176eedc77632c0d3dc7fed6776b9a"} err="failed to get container status \"f7ba0012e72fb97e77e1e784c962684f9ec176eedc77632c0d3dc7fed6776b9a\": rpc error: code = NotFound desc = could not find container \"f7ba0012e72fb97e77e1e784c962684f9ec176eedc77632c0d3dc7fed6776b9a\": container with ID starting with f7ba0012e72fb97e77e1e784c962684f9ec176eedc77632c0d3dc7fed6776b9a not found: ID does not exist" Sep 18 06:42:39.262626 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:39.262580 2567 scope.go:117] "RemoveContainer" containerID="10cdf5bf9e6413f6549226c2b82180e851cf6019c48b9dbbd547e0739245a804" Sep 18 06:42:39.262909 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:42:39.262889 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10cdf5bf9e6413f6549226c2b82180e851cf6019c48b9dbbd547e0739245a804\": container with ID starting with 10cdf5bf9e6413f6549226c2b82180e851cf6019c48b9dbbd547e0739245a804 not found: ID does not exist" containerID="10cdf5bf9e6413f6549226c2b82180e851cf6019c48b9dbbd547e0739245a804" Sep 18 06:42:39.262984 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:39.262922 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10cdf5bf9e6413f6549226c2b82180e851cf6019c48b9dbbd547e0739245a804"} err="failed to get container status \"10cdf5bf9e6413f6549226c2b82180e851cf6019c48b9dbbd547e0739245a804\": rpc error: code = NotFound desc = could not find container \"10cdf5bf9e6413f6549226c2b82180e851cf6019c48b9dbbd547e0739245a804\": container with ID starting with 10cdf5bf9e6413f6549226c2b82180e851cf6019c48b9dbbd547e0739245a804 not found: ID does not exist" Sep 18 06:42:39.266601 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:39.266581 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc"] Sep 18 06:42:39.270629 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:39.270608 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-5e4c6960-openshift-default-66f49f6f9b-hrncc"] Sep 18 06:42:40.459581 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:40.459542 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9aa5f688-009b-4232-8ec7-e007f8f7458d" path="/var/lib/kubelet/pods/9aa5f688-009b-4232-8ec7-e007f8f7458d/volumes" Sep 18 06:42:41.085772 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:41.085740 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-246af4f6-ccf5476c8-974q4"] Sep 18 06:42:41.086179 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:41.086167 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9aa5f688-009b-4232-8ec7-e007f8f7458d" containerName="istio-proxy" Sep 18 06:42:41.086239 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:41.086181 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="9aa5f688-009b-4232-8ec7-e007f8f7458d" containerName="istio-proxy" Sep 18 06:42:41.086239 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:41.086195 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9aa5f688-009b-4232-8ec7-e007f8f7458d" containerName="istio-proxy" Sep 18 06:42:41.086239 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:41.086201 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="9aa5f688-009b-4232-8ec7-e007f8f7458d" containerName="istio-proxy" Sep 18 06:42:41.086331 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:41.086269 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="9aa5f688-009b-4232-8ec7-e007f8f7458d" containerName="istio-proxy" Sep 18 06:42:41.086331 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:41.086277 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="9aa5f688-009b-4232-8ec7-e007f8f7458d" containerName="istio-proxy" Sep 18 06:42:41.090765 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:41.090746 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-246af4f6-ccf5476c8-974q4" Sep 18 06:42:41.093467 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:41.093440 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-246af4f6\"" Sep 18 06:42:41.093593 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:41.093466 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-246af4f6-dockercfg-b4wcg\"" Sep 18 06:42:41.099912 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:41.099892 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-246af4f6-ccf5476c8-974q4"] Sep 18 06:42:41.126398 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:41.126355 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-246af4f6-64b9cd8c8f-grqfh"] Sep 18 06:42:41.129681 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:41.129666 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-246af4f6-64b9cd8c8f-grqfh" Sep 18 06:42:41.140520 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:41.140490 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-246af4f6-64b9cd8c8f-grqfh"] Sep 18 06:42:41.239143 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:41.239104 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b38612cd-4913-481c-9052-a68ce412bfd0-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-246af4f6-ccf5476c8-974q4\" (UID: \"b38612cd-4913-481c-9052-a68ce412bfd0\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-246af4f6-ccf5476c8-974q4" Sep 18 06:42:41.239323 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:41.239206 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c0fe593-9c8a-4e19-8d35-42b14baaf1ba-plugins-config-volume\") pod \"payload-processing-e2e-derive-246af4f6-64b9cd8c8f-grqfh\" (UID: \"9c0fe593-9c8a-4e19-8d35-42b14baaf1ba\") " pod="openshift-ingress/payload-processing-e2e-derive-246af4f6-64b9cd8c8f-grqfh" Sep 18 06:42:41.239323 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:41.239244 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hpz6\" (UniqueName: \"kubernetes.io/projected/b38612cd-4913-481c-9052-a68ce412bfd0-kube-api-access-2hpz6\") pod \"payload-pre-processing-e2e-derive-246af4f6-ccf5476c8-974q4\" (UID: \"b38612cd-4913-481c-9052-a68ce412bfd0\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-246af4f6-ccf5476c8-974q4" Sep 18 06:42:41.239464 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:41.239346 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7nxz\" (UniqueName: \"kubernetes.io/projected/9c0fe593-9c8a-4e19-8d35-42b14baaf1ba-kube-api-access-s7nxz\") pod \"payload-processing-e2e-derive-246af4f6-64b9cd8c8f-grqfh\" (UID: \"9c0fe593-9c8a-4e19-8d35-42b14baaf1ba\") " pod="openshift-ingress/payload-processing-e2e-derive-246af4f6-64b9cd8c8f-grqfh" Sep 18 06:42:41.340274 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:41.340200 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7nxz\" (UniqueName: \"kubernetes.io/projected/9c0fe593-9c8a-4e19-8d35-42b14baaf1ba-kube-api-access-s7nxz\") pod \"payload-processing-e2e-derive-246af4f6-64b9cd8c8f-grqfh\" (UID: \"9c0fe593-9c8a-4e19-8d35-42b14baaf1ba\") " pod="openshift-ingress/payload-processing-e2e-derive-246af4f6-64b9cd8c8f-grqfh" Sep 18 06:42:41.340274 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:41.340273 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b38612cd-4913-481c-9052-a68ce412bfd0-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-246af4f6-ccf5476c8-974q4\" (UID: \"b38612cd-4913-481c-9052-a68ce412bfd0\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-246af4f6-ccf5476c8-974q4" Sep 18 06:42:41.340489 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:41.340389 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c0fe593-9c8a-4e19-8d35-42b14baaf1ba-plugins-config-volume\") pod \"payload-processing-e2e-derive-246af4f6-64b9cd8c8f-grqfh\" (UID: \"9c0fe593-9c8a-4e19-8d35-42b14baaf1ba\") " pod="openshift-ingress/payload-processing-e2e-derive-246af4f6-64b9cd8c8f-grqfh" Sep 18 06:42:41.340489 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:41.340428 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hpz6\" (UniqueName: \"kubernetes.io/projected/b38612cd-4913-481c-9052-a68ce412bfd0-kube-api-access-2hpz6\") pod \"payload-pre-processing-e2e-derive-246af4f6-ccf5476c8-974q4\" (UID: \"b38612cd-4913-481c-9052-a68ce412bfd0\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-246af4f6-ccf5476c8-974q4" Sep 18 06:42:41.341023 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:41.341001 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b38612cd-4913-481c-9052-a68ce412bfd0-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-246af4f6-ccf5476c8-974q4\" (UID: \"b38612cd-4913-481c-9052-a68ce412bfd0\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-246af4f6-ccf5476c8-974q4" Sep 18 06:42:41.341060 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:41.341018 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c0fe593-9c8a-4e19-8d35-42b14baaf1ba-plugins-config-volume\") pod \"payload-processing-e2e-derive-246af4f6-64b9cd8c8f-grqfh\" (UID: \"9c0fe593-9c8a-4e19-8d35-42b14baaf1ba\") " pod="openshift-ingress/payload-processing-e2e-derive-246af4f6-64b9cd8c8f-grqfh" Sep 18 06:42:41.347991 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:41.347968 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hpz6\" (UniqueName: \"kubernetes.io/projected/b38612cd-4913-481c-9052-a68ce412bfd0-kube-api-access-2hpz6\") pod \"payload-pre-processing-e2e-derive-246af4f6-ccf5476c8-974q4\" (UID: \"b38612cd-4913-481c-9052-a68ce412bfd0\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-246af4f6-ccf5476c8-974q4" Sep 18 06:42:41.348099 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:41.348010 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7nxz\" (UniqueName: \"kubernetes.io/projected/9c0fe593-9c8a-4e19-8d35-42b14baaf1ba-kube-api-access-s7nxz\") pod \"payload-processing-e2e-derive-246af4f6-64b9cd8c8f-grqfh\" (UID: \"9c0fe593-9c8a-4e19-8d35-42b14baaf1ba\") " pod="openshift-ingress/payload-processing-e2e-derive-246af4f6-64b9cd8c8f-grqfh" Sep 18 06:42:41.402129 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:41.402093 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-246af4f6-ccf5476c8-974q4" Sep 18 06:42:41.440199 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:41.440169 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-246af4f6-64b9cd8c8f-grqfh" Sep 18 06:42:41.674617 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:41.674589 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-246af4f6-ccf5476c8-974q4"] Sep 18 06:42:41.675701 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:42:41.675669 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb38612cd_4913_481c_9052_a68ce412bfd0.slice/crio-946a88c31bf65688b719d5640867179a830ae880ed5fd4d1fcc12a0ee974979a WatchSource:0}: Error finding container 946a88c31bf65688b719d5640867179a830ae880ed5fd4d1fcc12a0ee974979a: Status 404 returned error can't find the container with id 946a88c31bf65688b719d5640867179a830ae880ed5fd4d1fcc12a0ee974979a Sep 18 06:42:41.720747 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:41.720723 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-246af4f6-64b9cd8c8f-grqfh"] Sep 18 06:42:41.720849 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:42:41.720830 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c0fe593_9c8a_4e19_8d35_42b14baaf1ba.slice/crio-1fd9a4fdbe5b4eff50cb5efda1b6871e59e9a63cd71aecd4f327921b67d26a26 WatchSource:0}: Error finding container 1fd9a4fdbe5b4eff50cb5efda1b6871e59e9a63cd71aecd4f327921b67d26a26: Status 404 returned error can't find the container with id 1fd9a4fdbe5b4eff50cb5efda1b6871e59e9a63cd71aecd4f327921b67d26a26 Sep 18 06:42:42.264347 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:42.264314 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-246af4f6-ccf5476c8-974q4" event={"ID":"b38612cd-4913-481c-9052-a68ce412bfd0","Type":"ContainerStarted","Data":"42cc306c6a07db7314cabd7cf315d4c500ceceaab3622228c44718d99de5b782"} Sep 18 06:42:42.264576 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:42.264397 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-246af4f6-ccf5476c8-974q4" Sep 18 06:42:42.264576 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:42.264422 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-246af4f6-ccf5476c8-974q4" event={"ID":"b38612cd-4913-481c-9052-a68ce412bfd0","Type":"ContainerStarted","Data":"946a88c31bf65688b719d5640867179a830ae880ed5fd4d1fcc12a0ee974979a"} Sep 18 06:42:42.265938 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:42.265913 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-246af4f6-64b9cd8c8f-grqfh" event={"ID":"9c0fe593-9c8a-4e19-8d35-42b14baaf1ba","Type":"ContainerStarted","Data":"547eb42094aed037df9bc3f97f0eb70a4174a958c2d460a6eacc641fabab2839"} Sep 18 06:42:42.266039 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:42.265959 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-246af4f6-64b9cd8c8f-grqfh" event={"ID":"9c0fe593-9c8a-4e19-8d35-42b14baaf1ba","Type":"ContainerStarted","Data":"1fd9a4fdbe5b4eff50cb5efda1b6871e59e9a63cd71aecd4f327921b67d26a26"} Sep 18 06:42:42.266039 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:42.265987 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-246af4f6-64b9cd8c8f-grqfh" Sep 18 06:42:42.279483 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:42.279441 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-246af4f6-ccf5476c8-974q4" podStartSLOduration=1.279428415 podStartE2EDuration="1.279428415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:42:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:42:42.27879011 +0000 UTC m=+1224.018206326" watchObservedRunningTime="2026-09-18 06:42:42.279428415 +0000 UTC m=+1224.018844588" Sep 18 06:42:42.295207 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:42.295159 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-246af4f6-64b9cd8c8f-grqfh" podStartSLOduration=1.295146902 podStartE2EDuration="1.295146902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:42:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:42:42.293122799 +0000 UTC m=+1224.032538997" watchObservedRunningTime="2026-09-18 06:42:42.295146902 +0000 UTC m=+1224.034563075" Sep 18 06:42:53.274448 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:53.274417 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-246af4f6-64b9cd8c8f-grqfh" Sep 18 06:42:53.274930 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:53.274506 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-246af4f6-ccf5476c8-974q4" Sep 18 06:42:57.359940 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:57.359911 2567 generic.go:356] "Generic (PLEG): container finished" podID="67f1cb84-a6be-4c3d-bc42-0876725c58aa" containerID="5fb8e3c0deff4238e75f9b9584d25e506613feb4210d47c52ad032faf4c9260a" exitCode=0 Sep 18 06:42:57.360409 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:57.359989 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw" event={"ID":"67f1cb84-a6be-4c3d-bc42-0876725c58aa","Type":"ContainerDied","Data":"5fb8e3c0deff4238e75f9b9584d25e506613feb4210d47c52ad032faf4c9260a"} Sep 18 06:42:57.360409 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:57.360389 2567 scope.go:117] "RemoveContainer" containerID="5fb8e3c0deff4238e75f9b9584d25e506613feb4210d47c52ad032faf4c9260a" Sep 18 06:42:57.361508 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:57.361477 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:42:57.361567 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:57.361554 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:42:57.361602 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:57.361585 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:42:58.372382 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:58.372341 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw" event={"ID":"67f1cb84-a6be-4c3d-bc42-0876725c58aa","Type":"ContainerStarted","Data":"4812a0606a97ef4e3871b15b5dc12bf42d471e3c126a433ed855fdaec2a0f0c8"} Sep 18 06:42:59.200758 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:59.200725 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw" Sep 18 06:42:59.201011 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:59.200990 2567 patch_prober.go:28] interesting pod/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Sep 18 06:42:59.201078 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:42:59.201058 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw" podUID="67f1cb84-a6be-4c3d-bc42-0876725c58aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Sep 18 06:43:00.201790 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:00.201755 2567 patch_prober.go:28] interesting pod/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Sep 18 06:43:00.202183 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:00.201808 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw" podUID="67f1cb84-a6be-4c3d-bc42-0876725c58aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Sep 18 06:43:01.201440 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:01.201406 2567 patch_prober.go:28] interesting pod/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Sep 18 06:43:01.201616 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:01.201460 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw" podUID="67f1cb84-a6be-4c3d-bc42-0876725c58aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Sep 18 06:43:02.201987 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:02.201902 2567 patch_prober.go:28] interesting pod/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Sep 18 06:43:02.201987 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:02.201960 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw" podUID="67f1cb84-a6be-4c3d-bc42-0876725c58aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Sep 18 06:43:03.201557 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:03.201518 2567 patch_prober.go:28] interesting pod/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Sep 18 06:43:03.201740 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:03.201577 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw" podUID="67f1cb84-a6be-4c3d-bc42-0876725c58aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Sep 18 06:43:04.205445 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:04.205414 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw" Sep 18 06:43:04.205876 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:04.205713 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw" Sep 18 06:43:04.206458 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:04.206440 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw" Sep 18 06:43:06.457867 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:06.457835 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-246af4f6-87t6h"] Sep 18 06:43:06.460614 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:06.460594 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-246af4f6-87t6h" Sep 18 06:43:06.469484 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:06.468993 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-246af4f6-87t6h"] Sep 18 06:43:06.477905 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:06.477886 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-88t7h\"" Sep 18 06:43:06.478017 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:06.477919 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 18 06:43:06.579198 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:06.579173 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17997472-d105-4f38-9522-01a8f3ead08f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-246af4f6-87t6h\" (UID: \"17997472-d105-4f38-9522-01a8f3ead08f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-246af4f6-87t6h" Sep 18 06:43:06.680064 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:06.680034 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17997472-d105-4f38-9522-01a8f3ead08f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-246af4f6-87t6h\" (UID: \"17997472-d105-4f38-9522-01a8f3ead08f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-246af4f6-87t6h" Sep 18 06:43:06.680689 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:06.680664 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17997472-d105-4f38-9522-01a8f3ead08f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-246af4f6-87t6h\" (UID: \"17997472-d105-4f38-9522-01a8f3ead08f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-246af4f6-87t6h" Sep 18 06:43:06.774188 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:06.774116 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-246af4f6-87t6h" Sep 18 06:43:07.045656 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:07.045631 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-246af4f6-87t6h"] Sep 18 06:43:07.046613 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:43:07.046588 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17997472_d105_4f38_9522_01a8f3ead08f.slice/crio-4967ffd797ae13128eed642d80aec645b5f455aaa4ee2e498f3525715d4819be WatchSource:0}: Error finding container 4967ffd797ae13128eed642d80aec645b5f455aaa4ee2e498f3525715d4819be: Status 404 returned error can't find the container with id 4967ffd797ae13128eed642d80aec645b5f455aaa4ee2e498f3525715d4819be Sep 18 06:43:07.429350 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:07.429319 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-246af4f6-87t6h" event={"ID":"17997472-d105-4f38-9522-01a8f3ead08f","Type":"ContainerStarted","Data":"bcb7645953980389599742ce343c881b5fa5270a3f08651004fa2084f6e9d111"} Sep 18 06:43:07.429532 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:07.429393 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-246af4f6-87t6h" event={"ID":"17997472-d105-4f38-9522-01a8f3ead08f","Type":"ContainerStarted","Data":"4967ffd797ae13128eed642d80aec645b5f455aaa4ee2e498f3525715d4819be"} Sep 18 06:43:07.446775 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:07.446727 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-246af4f6-87t6h" podStartSLOduration=1.4467113010000001 podStartE2EDuration="1.446711301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:43:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:43:07.443831759 +0000 UTC m=+1249.183247959" watchObservedRunningTime="2026-09-18 06:43:07.446711301 +0000 UTC m=+1249.186127475" Sep 18 06:43:08.438000 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:08.437965 2567 generic.go:356] "Generic (PLEG): container finished" podID="17997472-d105-4f38-9522-01a8f3ead08f" containerID="bcb7645953980389599742ce343c881b5fa5270a3f08651004fa2084f6e9d111" exitCode=0 Sep 18 06:43:08.438460 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:08.438047 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-246af4f6-87t6h" event={"ID":"17997472-d105-4f38-9522-01a8f3ead08f","Type":"ContainerDied","Data":"bcb7645953980389599742ce343c881b5fa5270a3f08651004fa2084f6e9d111"} Sep 18 06:43:09.583303 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:09.583279 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-246af4f6-87t6h" Sep 18 06:43:09.708716 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:09.708652 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17997472-d105-4f38-9522-01a8f3ead08f-maas-api-service-ca\") pod \"17997472-d105-4f38-9522-01a8f3ead08f\" (UID: \"17997472-d105-4f38-9522-01a8f3ead08f\") " Sep 18 06:43:09.708990 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:09.708970 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17997472-d105-4f38-9522-01a8f3ead08f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "17997472-d105-4f38-9522-01a8f3ead08f" (UID: "17997472-d105-4f38-9522-01a8f3ead08f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 06:43:09.809383 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:09.809339 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17997472-d105-4f38-9522-01a8f3ead08f-maas-api-service-ca\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:43:10.454224 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:10.454193 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-246af4f6-87t6h" Sep 18 06:43:10.454224 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:10.454208 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-246af4f6-87t6h" event={"ID":"17997472-d105-4f38-9522-01a8f3ead08f","Type":"ContainerDied","Data":"4967ffd797ae13128eed642d80aec645b5f455aaa4ee2e498f3525715d4819be"} Sep 18 06:43:10.454500 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:10.454252 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4967ffd797ae13128eed642d80aec645b5f455aaa4ee2e498f3525715d4819be" Sep 18 06:43:18.662206 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:18.662173 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-246af4f6-64b9cd8c8f-grqfh"] Sep 18 06:43:18.662731 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:18.662448 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-246af4f6-64b9cd8c8f-grqfh" podUID="9c0fe593-9c8a-4e19-8d35-42b14baaf1ba" containerName="payload-processing" containerID="cri-o://547eb42094aed037df9bc3f97f0eb70a4174a958c2d460a6eacc641fabab2839" gracePeriod=30 Sep 18 06:43:18.668145 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:18.668116 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-246af4f6-ccf5476c8-974q4"] Sep 18 06:43:18.668560 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:18.668510 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-246af4f6-ccf5476c8-974q4" podUID="b38612cd-4913-481c-9052-a68ce412bfd0" containerName="payload-pre-processing" containerID="cri-o://42cc306c6a07db7314cabd7cf315d4c500ceceaab3622228c44718d99de5b782" gracePeriod=30 Sep 18 06:43:18.811208 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:18.811146 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-246af4f6-87t6h"] Sep 18 06:43:18.814178 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:18.814150 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-246af4f6-87t6h"] Sep 18 06:43:18.953257 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:18.952713 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw"] Sep 18 06:43:18.953257 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:18.952962 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw" podUID="67f1cb84-a6be-4c3d-bc42-0876725c58aa" containerName="istio-proxy" containerID="cri-o://4812a0606a97ef4e3871b15b5dc12bf42d471e3c126a433ed855fdaec2a0f0c8" gracePeriod=30 Sep 18 06:43:18.975740 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:18.975718 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-246af4f6-64b9cd8c8f-grqfh" Sep 18 06:43:18.980490 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:18.980467 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-246af4f6-ccf5476c8-974q4" Sep 18 06:43:19.092802 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:19.092771 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b38612cd-4913-481c-9052-a68ce412bfd0-plugins-config-volume\") pod \"b38612cd-4913-481c-9052-a68ce412bfd0\" (UID: \"b38612cd-4913-481c-9052-a68ce412bfd0\") " Sep 18 06:43:19.092975 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:19.092840 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2hpz6\" (UniqueName: \"kubernetes.io/projected/b38612cd-4913-481c-9052-a68ce412bfd0-kube-api-access-2hpz6\") pod \"b38612cd-4913-481c-9052-a68ce412bfd0\" (UID: \"b38612cd-4913-481c-9052-a68ce412bfd0\") " Sep 18 06:43:19.092975 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:19.092868 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c0fe593-9c8a-4e19-8d35-42b14baaf1ba-plugins-config-volume\") pod \"9c0fe593-9c8a-4e19-8d35-42b14baaf1ba\" (UID: \"9c0fe593-9c8a-4e19-8d35-42b14baaf1ba\") " Sep 18 06:43:19.092975 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:19.092889 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s7nxz\" (UniqueName: \"kubernetes.io/projected/9c0fe593-9c8a-4e19-8d35-42b14baaf1ba-kube-api-access-s7nxz\") pod \"9c0fe593-9c8a-4e19-8d35-42b14baaf1ba\" (UID: \"9c0fe593-9c8a-4e19-8d35-42b14baaf1ba\") " Sep 18 06:43:19.093236 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:19.093206 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b38612cd-4913-481c-9052-a68ce412bfd0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b38612cd-4913-481c-9052-a68ce412bfd0" (UID: "b38612cd-4913-481c-9052-a68ce412bfd0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 06:43:19.093303 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:19.093207 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c0fe593-9c8a-4e19-8d35-42b14baaf1ba-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9c0fe593-9c8a-4e19-8d35-42b14baaf1ba" (UID: "9c0fe593-9c8a-4e19-8d35-42b14baaf1ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 06:43:19.095034 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:19.095010 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c0fe593-9c8a-4e19-8d35-42b14baaf1ba-kube-api-access-s7nxz" (OuterVolumeSpecName: "kube-api-access-s7nxz") pod "9c0fe593-9c8a-4e19-8d35-42b14baaf1ba" (UID: "9c0fe593-9c8a-4e19-8d35-42b14baaf1ba"). InnerVolumeSpecName "kube-api-access-s7nxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 06:43:19.095129 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:19.095053 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b38612cd-4913-481c-9052-a68ce412bfd0-kube-api-access-2hpz6" (OuterVolumeSpecName: "kube-api-access-2hpz6") pod "b38612cd-4913-481c-9052-a68ce412bfd0" (UID: "b38612cd-4913-481c-9052-a68ce412bfd0"). InnerVolumeSpecName "kube-api-access-2hpz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 06:43:19.194118 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:19.194044 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7nxz\" (UniqueName: \"kubernetes.io/projected/9c0fe593-9c8a-4e19-8d35-42b14baaf1ba-kube-api-access-s7nxz\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:43:19.194118 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:19.194067 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b38612cd-4913-481c-9052-a68ce412bfd0-plugins-config-volume\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:43:19.194118 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:19.194077 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hpz6\" (UniqueName: \"kubernetes.io/projected/b38612cd-4913-481c-9052-a68ce412bfd0-kube-api-access-2hpz6\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:43:19.194118 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:19.194086 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c0fe593-9c8a-4e19-8d35-42b14baaf1ba-plugins-config-volume\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:43:19.207706 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:19.207676 2567 patch_prober.go:28] interesting pod/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 18 06:43:19.207812 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:19.207735 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw" podUID="67f1cb84-a6be-4c3d-bc42-0876725c58aa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 18 06:43:19.506123 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:19.506036 2567 generic.go:356] "Generic (PLEG): container finished" podID="b38612cd-4913-481c-9052-a68ce412bfd0" containerID="42cc306c6a07db7314cabd7cf315d4c500ceceaab3622228c44718d99de5b782" exitCode=0 Sep 18 06:43:19.506123 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:19.506080 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-246af4f6-ccf5476c8-974q4" Sep 18 06:43:19.506123 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:19.506097 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-246af4f6-ccf5476c8-974q4" event={"ID":"b38612cd-4913-481c-9052-a68ce412bfd0","Type":"ContainerDied","Data":"42cc306c6a07db7314cabd7cf315d4c500ceceaab3622228c44718d99de5b782"} Sep 18 06:43:19.506402 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:19.506152 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-246af4f6-ccf5476c8-974q4" event={"ID":"b38612cd-4913-481c-9052-a68ce412bfd0","Type":"ContainerDied","Data":"946a88c31bf65688b719d5640867179a830ae880ed5fd4d1fcc12a0ee974979a"} Sep 18 06:43:19.506402 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:19.506182 2567 scope.go:117] "RemoveContainer" containerID="42cc306c6a07db7314cabd7cf315d4c500ceceaab3622228c44718d99de5b782" Sep 18 06:43:19.507646 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:19.507622 2567 generic.go:356] "Generic (PLEG): container finished" podID="9c0fe593-9c8a-4e19-8d35-42b14baaf1ba" containerID="547eb42094aed037df9bc3f97f0eb70a4174a958c2d460a6eacc641fabab2839" exitCode=0 Sep 18 06:43:19.507745 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:19.507673 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-246af4f6-64b9cd8c8f-grqfh" Sep 18 06:43:19.507745 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:19.507688 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-246af4f6-64b9cd8c8f-grqfh" event={"ID":"9c0fe593-9c8a-4e19-8d35-42b14baaf1ba","Type":"ContainerDied","Data":"547eb42094aed037df9bc3f97f0eb70a4174a958c2d460a6eacc641fabab2839"} Sep 18 06:43:19.507745 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:19.507739 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-246af4f6-64b9cd8c8f-grqfh" event={"ID":"9c0fe593-9c8a-4e19-8d35-42b14baaf1ba","Type":"ContainerDied","Data":"1fd9a4fdbe5b4eff50cb5efda1b6871e59e9a63cd71aecd4f327921b67d26a26"} Sep 18 06:43:19.516214 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:19.516198 2567 scope.go:117] "RemoveContainer" containerID="42cc306c6a07db7314cabd7cf315d4c500ceceaab3622228c44718d99de5b782" Sep 18 06:43:19.516557 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:43:19.516538 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42cc306c6a07db7314cabd7cf315d4c500ceceaab3622228c44718d99de5b782\": container with ID starting with 42cc306c6a07db7314cabd7cf315d4c500ceceaab3622228c44718d99de5b782 not found: ID does not exist" containerID="42cc306c6a07db7314cabd7cf315d4c500ceceaab3622228c44718d99de5b782" Sep 18 06:43:19.516634 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:19.516566 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42cc306c6a07db7314cabd7cf315d4c500ceceaab3622228c44718d99de5b782"} err="failed to get container status \"42cc306c6a07db7314cabd7cf315d4c500ceceaab3622228c44718d99de5b782\": rpc error: code = NotFound desc = could not find container \"42cc306c6a07db7314cabd7cf315d4c500ceceaab3622228c44718d99de5b782\": container with ID starting with 42cc306c6a07db7314cabd7cf315d4c500ceceaab3622228c44718d99de5b782 not found: ID does not exist" Sep 18 06:43:19.516634 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:19.516587 2567 scope.go:117] "RemoveContainer" containerID="547eb42094aed037df9bc3f97f0eb70a4174a958c2d460a6eacc641fabab2839" Sep 18 06:43:19.525772 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:19.525753 2567 scope.go:117] "RemoveContainer" containerID="547eb42094aed037df9bc3f97f0eb70a4174a958c2d460a6eacc641fabab2839" Sep 18 06:43:19.526029 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:43:19.526011 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"547eb42094aed037df9bc3f97f0eb70a4174a958c2d460a6eacc641fabab2839\": container with ID starting with 547eb42094aed037df9bc3f97f0eb70a4174a958c2d460a6eacc641fabab2839 not found: ID does not exist" containerID="547eb42094aed037df9bc3f97f0eb70a4174a958c2d460a6eacc641fabab2839" Sep 18 06:43:19.526076 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:19.526036 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"547eb42094aed037df9bc3f97f0eb70a4174a958c2d460a6eacc641fabab2839"} err="failed to get container status \"547eb42094aed037df9bc3f97f0eb70a4174a958c2d460a6eacc641fabab2839\": rpc error: code = NotFound desc = could not find container \"547eb42094aed037df9bc3f97f0eb70a4174a958c2d460a6eacc641fabab2839\": container with ID starting with 547eb42094aed037df9bc3f97f0eb70a4174a958c2d460a6eacc641fabab2839 not found: ID does not exist" Sep 18 06:43:19.532612 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:19.532590 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-246af4f6-64b9cd8c8f-grqfh"] Sep 18 06:43:19.535799 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:19.535779 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-246af4f6-64b9cd8c8f-grqfh"] Sep 18 06:43:19.548331 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:19.548311 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-246af4f6-ccf5476c8-974q4"] Sep 18 06:43:19.551852 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:19.551832 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-246af4f6-ccf5476c8-974q4"] Sep 18 06:43:20.447254 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:20.447219 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17997472-d105-4f38-9522-01a8f3ead08f" path="/var/lib/kubelet/pods/17997472-d105-4f38-9522-01a8f3ead08f/volumes" Sep 18 06:43:20.447640 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:20.447626 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c0fe593-9c8a-4e19-8d35-42b14baaf1ba" path="/var/lib/kubelet/pods/9c0fe593-9c8a-4e19-8d35-42b14baaf1ba/volumes" Sep 18 06:43:20.447942 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:20.447930 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b38612cd-4913-481c-9052-a68ce412bfd0" path="/var/lib/kubelet/pods/b38612cd-4913-481c-9052-a68ce412bfd0/volumes" Sep 18 06:43:23.910434 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:23.910405 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-43fdfe-j5v52"] Sep 18 06:43:23.910839 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:23.910801 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b38612cd-4913-481c-9052-a68ce412bfd0" containerName="payload-pre-processing" Sep 18 06:43:23.910839 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:23.910813 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b38612cd-4913-481c-9052-a68ce412bfd0" containerName="payload-pre-processing" Sep 18 06:43:23.910839 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:23.910827 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c0fe593-9c8a-4e19-8d35-42b14baaf1ba" containerName="payload-processing" Sep 18 06:43:23.910839 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:23.910833 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c0fe593-9c8a-4e19-8d35-42b14baaf1ba" containerName="payload-processing" Sep 18 06:43:23.910839 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:23.910842 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17997472-d105-4f38-9522-01a8f3ead08f" containerName="revoke-keys" Sep 18 06:43:23.911020 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:23.910847 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="17997472-d105-4f38-9522-01a8f3ead08f" containerName="revoke-keys" Sep 18 06:43:23.911020 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:23.910919 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c0fe593-9c8a-4e19-8d35-42b14baaf1ba" containerName="payload-processing" Sep 18 06:43:23.911020 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:23.910928 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="17997472-d105-4f38-9522-01a8f3ead08f" containerName="revoke-keys" Sep 18 06:43:23.911020 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:23.910937 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="b38612cd-4913-481c-9052-a68ce412bfd0" containerName="payload-pre-processing" Sep 18 06:43:23.913850 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:23.913834 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-43fdfe-j5v52" Sep 18 06:43:23.917634 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:23.917611 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 18 06:43:23.917865 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:23.917694 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-88t7h\"" Sep 18 06:43:23.919816 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:23.919793 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-43fdfe-j5v52"] Sep 18 06:43:24.044310 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:24.044280 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30aa103a-8c8e-4ee2-af67-0938b4224da4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-43fdfe-j5v52\" (UID: \"30aa103a-8c8e-4ee2-af67-0938b4224da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-43fdfe-j5v52" Sep 18 06:43:24.145708 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:24.145677 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30aa103a-8c8e-4ee2-af67-0938b4224da4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-43fdfe-j5v52\" (UID: \"30aa103a-8c8e-4ee2-af67-0938b4224da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-43fdfe-j5v52" Sep 18 06:43:24.146321 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:24.146300 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30aa103a-8c8e-4ee2-af67-0938b4224da4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-43fdfe-j5v52\" (UID: \"30aa103a-8c8e-4ee2-af67-0938b4224da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-43fdfe-j5v52" Sep 18 06:43:24.208546 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:24.208524 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw" Sep 18 06:43:24.227463 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:24.227432 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-43fdfe-j5v52" Sep 18 06:43:24.346880 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:24.346839 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f1cb84-a6be-4c3d-bc42-0876725c58aa-credential-socket\") pod \"67f1cb84-a6be-4c3d-bc42-0876725c58aa\" (UID: \"67f1cb84-a6be-4c3d-bc42-0876725c58aa\") " Sep 18 06:43:24.346880 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:24.346878 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f1cb84-a6be-4c3d-bc42-0876725c58aa-workload-socket\") pod \"67f1cb84-a6be-4c3d-bc42-0876725c58aa\" (UID: \"67f1cb84-a6be-4c3d-bc42-0876725c58aa\") " Sep 18 06:43:24.347116 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:24.346913 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67f1cb84-a6be-4c3d-bc42-0876725c58aa-istiod-ca-cert\") pod \"67f1cb84-a6be-4c3d-bc42-0876725c58aa\" (UID: \"67f1cb84-a6be-4c3d-bc42-0876725c58aa\") " Sep 18 06:43:24.347116 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:24.346950 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67f1cb84-a6be-4c3d-bc42-0876725c58aa-istio-token\") pod \"67f1cb84-a6be-4c3d-bc42-0876725c58aa\" (UID: \"67f1cb84-a6be-4c3d-bc42-0876725c58aa\") " Sep 18 06:43:24.347116 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:24.346974 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67f1cb84-a6be-4c3d-bc42-0876725c58aa-istio-envoy\") pod \"67f1cb84-a6be-4c3d-bc42-0876725c58aa\" (UID: \"67f1cb84-a6be-4c3d-bc42-0876725c58aa\") " Sep 18 06:43:24.347276 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:24.347133 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6pl4v\" (UniqueName: \"kubernetes.io/projected/67f1cb84-a6be-4c3d-bc42-0876725c58aa-kube-api-access-6pl4v\") pod \"67f1cb84-a6be-4c3d-bc42-0876725c58aa\" (UID: \"67f1cb84-a6be-4c3d-bc42-0876725c58aa\") " Sep 18 06:43:24.347276 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:24.347150 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67f1cb84-a6be-4c3d-bc42-0876725c58aa-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "67f1cb84-a6be-4c3d-bc42-0876725c58aa" (UID: "67f1cb84-a6be-4c3d-bc42-0876725c58aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:43:24.347276 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:24.347173 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67f1cb84-a6be-4c3d-bc42-0876725c58aa-istio-data\") pod \"67f1cb84-a6be-4c3d-bc42-0876725c58aa\" (UID: \"67f1cb84-a6be-4c3d-bc42-0876725c58aa\") " Sep 18 06:43:24.347276 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:24.347242 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67f1cb84-a6be-4c3d-bc42-0876725c58aa-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "67f1cb84-a6be-4c3d-bc42-0876725c58aa" (UID: "67f1cb84-a6be-4c3d-bc42-0876725c58aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:43:24.347276 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:24.347262 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67f1cb84-a6be-4c3d-bc42-0876725c58aa-workload-certs\") pod \"67f1cb84-a6be-4c3d-bc42-0876725c58aa\" (UID: \"67f1cb84-a6be-4c3d-bc42-0876725c58aa\") " Sep 18 06:43:24.347581 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:24.347296 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67f1cb84-a6be-4c3d-bc42-0876725c58aa-istio-podinfo\") pod \"67f1cb84-a6be-4c3d-bc42-0876725c58aa\" (UID: \"67f1cb84-a6be-4c3d-bc42-0876725c58aa\") " Sep 18 06:43:24.347581 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:24.347340 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67f1cb84-a6be-4c3d-bc42-0876725c58aa-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "67f1cb84-a6be-4c3d-bc42-0876725c58aa" (UID: "67f1cb84-a6be-4c3d-bc42-0876725c58aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 06:43:24.347581 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:24.347479 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67f1cb84-a6be-4c3d-bc42-0876725c58aa-istio-data" (OuterVolumeSpecName: "istio-data") pod "67f1cb84-a6be-4c3d-bc42-0876725c58aa" (UID: "67f1cb84-a6be-4c3d-bc42-0876725c58aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:43:24.347581 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:24.347524 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67f1cb84-a6be-4c3d-bc42-0876725c58aa-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "67f1cb84-a6be-4c3d-bc42-0876725c58aa" (UID: "67f1cb84-a6be-4c3d-bc42-0876725c58aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:43:24.347787 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:24.347744 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f1cb84-a6be-4c3d-bc42-0876725c58aa-credential-socket\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:43:24.347787 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:24.347764 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f1cb84-a6be-4c3d-bc42-0876725c58aa-workload-socket\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:43:24.347787 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:24.347779 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67f1cb84-a6be-4c3d-bc42-0876725c58aa-istiod-ca-cert\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:43:24.347942 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:24.347793 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67f1cb84-a6be-4c3d-bc42-0876725c58aa-istio-data\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:43:24.347942 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:24.347805 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67f1cb84-a6be-4c3d-bc42-0876725c58aa-workload-certs\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:43:24.349301 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:24.349271 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67f1cb84-a6be-4c3d-bc42-0876725c58aa-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "67f1cb84-a6be-4c3d-bc42-0876725c58aa" (UID: "67f1cb84-a6be-4c3d-bc42-0876725c58aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:43:24.349426 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:24.349377 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67f1cb84-a6be-4c3d-bc42-0876725c58aa-istio-token" (OuterVolumeSpecName: "istio-token") pod "67f1cb84-a6be-4c3d-bc42-0876725c58aa" (UID: "67f1cb84-a6be-4c3d-bc42-0876725c58aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 06:43:24.349426 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:24.349357 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67f1cb84-a6be-4c3d-bc42-0876725c58aa-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "67f1cb84-a6be-4c3d-bc42-0876725c58aa" (UID: "67f1cb84-a6be-4c3d-bc42-0876725c58aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 06:43:24.349426 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:24.349389 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67f1cb84-a6be-4c3d-bc42-0876725c58aa-kube-api-access-6pl4v" (OuterVolumeSpecName: "kube-api-access-6pl4v") pod "67f1cb84-a6be-4c3d-bc42-0876725c58aa" (UID: "67f1cb84-a6be-4c3d-bc42-0876725c58aa"). InnerVolumeSpecName "kube-api-access-6pl4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 06:43:24.448831 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:24.448747 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67f1cb84-a6be-4c3d-bc42-0876725c58aa-istio-podinfo\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:43:24.448831 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:24.448776 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67f1cb84-a6be-4c3d-bc42-0876725c58aa-istio-token\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:43:24.448831 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:24.448788 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67f1cb84-a6be-4c3d-bc42-0876725c58aa-istio-envoy\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:43:24.448831 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:24.448799 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pl4v\" (UniqueName: \"kubernetes.io/projected/67f1cb84-a6be-4c3d-bc42-0876725c58aa-kube-api-access-6pl4v\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:43:24.502555 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:24.502529 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-43fdfe-j5v52"] Sep 18 06:43:24.503602 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:43:24.503575 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30aa103a_8c8e_4ee2_af67_0938b4224da4.slice/crio-618625bc63245d006a75c4d81e949afefc7c963405b7cfcb2e5ef7e0847cfd35 WatchSource:0}: Error finding container 618625bc63245d006a75c4d81e949afefc7c963405b7cfcb2e5ef7e0847cfd35: Status 404 returned error can't find the container with id 618625bc63245d006a75c4d81e949afefc7c963405b7cfcb2e5ef7e0847cfd35 Sep 18 06:43:24.539928 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:24.539901 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-43fdfe-j5v52" event={"ID":"30aa103a-8c8e-4ee2-af67-0938b4224da4","Type":"ContainerStarted","Data":"618625bc63245d006a75c4d81e949afefc7c963405b7cfcb2e5ef7e0847cfd35"} Sep 18 06:43:24.541668 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:24.541647 2567 generic.go:356] "Generic (PLEG): container finished" podID="67f1cb84-a6be-4c3d-bc42-0876725c58aa" containerID="4812a0606a97ef4e3871b15b5dc12bf42d471e3c126a433ed855fdaec2a0f0c8" exitCode=0 Sep 18 06:43:24.541769 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:24.541680 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw" event={"ID":"67f1cb84-a6be-4c3d-bc42-0876725c58aa","Type":"ContainerDied","Data":"4812a0606a97ef4e3871b15b5dc12bf42d471e3c126a433ed855fdaec2a0f0c8"} Sep 18 06:43:24.541769 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:24.541691 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw" Sep 18 06:43:24.541769 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:24.541711 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw" event={"ID":"67f1cb84-a6be-4c3d-bc42-0876725c58aa","Type":"ContainerDied","Data":"58f46f9f19ee123b3616ebda5ae8e752a8ed33cb6e5fd1d79fd2c7a5a664b353"} Sep 18 06:43:24.541769 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:24.541731 2567 scope.go:117] "RemoveContainer" containerID="4812a0606a97ef4e3871b15b5dc12bf42d471e3c126a433ed855fdaec2a0f0c8" Sep 18 06:43:24.550978 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:24.550949 2567 scope.go:117] "RemoveContainer" containerID="5fb8e3c0deff4238e75f9b9584d25e506613feb4210d47c52ad032faf4c9260a" Sep 18 06:43:24.560010 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:24.559987 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw"] Sep 18 06:43:24.562100 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:24.562083 2567 scope.go:117] "RemoveContainer" containerID="4812a0606a97ef4e3871b15b5dc12bf42d471e3c126a433ed855fdaec2a0f0c8" Sep 18 06:43:24.562427 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:43:24.562410 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4812a0606a97ef4e3871b15b5dc12bf42d471e3c126a433ed855fdaec2a0f0c8\": container with ID starting with 4812a0606a97ef4e3871b15b5dc12bf42d471e3c126a433ed855fdaec2a0f0c8 not found: ID does not exist" containerID="4812a0606a97ef4e3871b15b5dc12bf42d471e3c126a433ed855fdaec2a0f0c8" Sep 18 06:43:24.562485 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:24.562435 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4812a0606a97ef4e3871b15b5dc12bf42d471e3c126a433ed855fdaec2a0f0c8"} err="failed to get container status \"4812a0606a97ef4e3871b15b5dc12bf42d471e3c126a433ed855fdaec2a0f0c8\": rpc error: code = NotFound desc = could not find container \"4812a0606a97ef4e3871b15b5dc12bf42d471e3c126a433ed855fdaec2a0f0c8\": container with ID starting with 4812a0606a97ef4e3871b15b5dc12bf42d471e3c126a433ed855fdaec2a0f0c8 not found: ID does not exist" Sep 18 06:43:24.562485 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:24.562453 2567 scope.go:117] "RemoveContainer" containerID="5fb8e3c0deff4238e75f9b9584d25e506613feb4210d47c52ad032faf4c9260a" Sep 18 06:43:24.562730 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:43:24.562713 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fb8e3c0deff4238e75f9b9584d25e506613feb4210d47c52ad032faf4c9260a\": container with ID starting with 5fb8e3c0deff4238e75f9b9584d25e506613feb4210d47c52ad032faf4c9260a not found: ID does not exist" containerID="5fb8e3c0deff4238e75f9b9584d25e506613feb4210d47c52ad032faf4c9260a" Sep 18 06:43:24.562771 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:24.562736 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fb8e3c0deff4238e75f9b9584d25e506613feb4210d47c52ad032faf4c9260a"} err="failed to get container status \"5fb8e3c0deff4238e75f9b9584d25e506613feb4210d47c52ad032faf4c9260a\": rpc error: code = NotFound desc = could not find container \"5fb8e3c0deff4238e75f9b9584d25e506613feb4210d47c52ad032faf4c9260a\": container with ID starting with 5fb8e3c0deff4238e75f9b9584d25e506613feb4210d47c52ad032faf4c9260a not found: ID does not exist" Sep 18 06:43:24.563458 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:24.563439 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-246af4f6-openshift-default-6876dc689b-gkppw"] Sep 18 06:43:25.551124 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:25.551040 2567 generic.go:356] "Generic (PLEG): container finished" podID="30aa103a-8c8e-4ee2-af67-0938b4224da4" containerID="2d3292fff86cfe2f358042a205b19b2e09d586ddfe4b2d0709783624ce569d20" exitCode=0 Sep 18 06:43:25.551124 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:25.551086 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-43fdfe-j5v52" event={"ID":"30aa103a-8c8e-4ee2-af67-0938b4224da4","Type":"ContainerDied","Data":"2d3292fff86cfe2f358042a205b19b2e09d586ddfe4b2d0709783624ce569d20"} Sep 18 06:43:26.446838 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:26.446807 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67f1cb84-a6be-4c3d-bc42-0876725c58aa" path="/var/lib/kubelet/pods/67f1cb84-a6be-4c3d-bc42-0876725c58aa/volumes" Sep 18 06:43:26.693549 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:26.693525 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-43fdfe-j5v52" Sep 18 06:43:26.770881 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:26.770796 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30aa103a-8c8e-4ee2-af67-0938b4224da4-maas-api-service-ca\") pod \"30aa103a-8c8e-4ee2-af67-0938b4224da4\" (UID: \"30aa103a-8c8e-4ee2-af67-0938b4224da4\") " Sep 18 06:43:26.771223 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:26.771198 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30aa103a-8c8e-4ee2-af67-0938b4224da4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "30aa103a-8c8e-4ee2-af67-0938b4224da4" (UID: "30aa103a-8c8e-4ee2-af67-0938b4224da4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 06:43:26.872248 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:26.872209 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30aa103a-8c8e-4ee2-af67-0938b4224da4-maas-api-service-ca\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:43:27.569825 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:27.569795 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-43fdfe-j5v52" Sep 18 06:43:27.569999 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:27.569836 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-43fdfe-j5v52" event={"ID":"30aa103a-8c8e-4ee2-af67-0938b4224da4","Type":"ContainerDied","Data":"618625bc63245d006a75c4d81e949afefc7c963405b7cfcb2e5ef7e0847cfd35"} Sep 18 06:43:27.569999 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:27.569882 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="618625bc63245d006a75c4d81e949afefc7c963405b7cfcb2e5ef7e0847cfd35" Sep 18 06:43:34.365261 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:34.365228 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-43fdfe-6bbcb85f7-wv7k8"] Sep 18 06:43:34.365904 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:34.365483 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-43fdfe-6bbcb85f7-wv7k8" podUID="80a7d765-9b80-49f8-8d64-f72ec2acb5d7" containerName="payload-processing" containerID="cri-o://0f6624657c7ba926e2913082c5905d8b5d88737d56d578492fff53bb4ef86381" gracePeriod=30 Sep 18 06:43:34.457710 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:34.457671 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-autores-43fdfe-6bbcb85f7-wv7k8" podUID="80a7d765-9b80-49f8-8d64-f72ec2acb5d7" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.50:9004: connect: connection refused" Sep 18 06:43:34.508576 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:34.508546 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-43fdfe-j5v52"] Sep 18 06:43:34.513708 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:34.513682 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-43fdfe-j5v52"] Sep 18 06:43:34.615314 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:34.615158 2567 generic.go:356] "Generic (PLEG): container finished" podID="80a7d765-9b80-49f8-8d64-f72ec2acb5d7" containerID="0f6624657c7ba926e2913082c5905d8b5d88737d56d578492fff53bb4ef86381" exitCode=0 Sep 18 06:43:34.615314 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:34.615185 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-43fdfe-6bbcb85f7-wv7k8" event={"ID":"80a7d765-9b80-49f8-8d64-f72ec2acb5d7","Type":"ContainerDied","Data":"0f6624657c7ba926e2913082c5905d8b5d88737d56d578492fff53bb4ef86381"} Sep 18 06:43:34.639834 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:34.639808 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-43fdfe-6bbcb85f7-wv7k8" Sep 18 06:43:34.743793 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:34.743762 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-knk25\" (UniqueName: \"kubernetes.io/projected/80a7d765-9b80-49f8-8d64-f72ec2acb5d7-kube-api-access-knk25\") pod \"80a7d765-9b80-49f8-8d64-f72ec2acb5d7\" (UID: \"80a7d765-9b80-49f8-8d64-f72ec2acb5d7\") " Sep 18 06:43:34.743948 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:34.743858 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80a7d765-9b80-49f8-8d64-f72ec2acb5d7-plugins-config-volume\") pod \"80a7d765-9b80-49f8-8d64-f72ec2acb5d7\" (UID: \"80a7d765-9b80-49f8-8d64-f72ec2acb5d7\") " Sep 18 06:43:34.744193 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:34.744173 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80a7d765-9b80-49f8-8d64-f72ec2acb5d7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "80a7d765-9b80-49f8-8d64-f72ec2acb5d7" (UID: "80a7d765-9b80-49f8-8d64-f72ec2acb5d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 06:43:34.744262 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:34.744251 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80a7d765-9b80-49f8-8d64-f72ec2acb5d7-plugins-config-volume\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:43:34.745951 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:34.745931 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80a7d765-9b80-49f8-8d64-f72ec2acb5d7-kube-api-access-knk25" (OuterVolumeSpecName: "kube-api-access-knk25") pod "80a7d765-9b80-49f8-8d64-f72ec2acb5d7" (UID: "80a7d765-9b80-49f8-8d64-f72ec2acb5d7"). InnerVolumeSpecName "kube-api-access-knk25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 06:43:34.845072 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:34.845047 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knk25\" (UniqueName: \"kubernetes.io/projected/80a7d765-9b80-49f8-8d64-f72ec2acb5d7-kube-api-access-knk25\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:43:35.623321 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:35.623286 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-43fdfe-6bbcb85f7-wv7k8" event={"ID":"80a7d765-9b80-49f8-8d64-f72ec2acb5d7","Type":"ContainerDied","Data":"f17ed739bc569d33cdc437e760dd7b9e5c57ed6dca1b1e210e9b1e733a2590ed"} Sep 18 06:43:35.623797 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:35.623306 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-43fdfe-6bbcb85f7-wv7k8" Sep 18 06:43:35.623797 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:35.623517 2567 scope.go:117] "RemoveContainer" containerID="0f6624657c7ba926e2913082c5905d8b5d88737d56d578492fff53bb4ef86381" Sep 18 06:43:35.645812 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:35.645789 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-43fdfe-6bbcb85f7-wv7k8"] Sep 18 06:43:35.648209 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:35.648188 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-43fdfe-6bbcb85f7-wv7k8"] Sep 18 06:43:36.445843 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:36.445813 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30aa103a-8c8e-4ee2-af67-0938b4224da4" path="/var/lib/kubelet/pods/30aa103a-8c8e-4ee2-af67-0938b4224da4/volumes" Sep 18 06:43:36.446222 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:43:36.446207 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80a7d765-9b80-49f8-8d64-f72ec2acb5d7" path="/var/lib/kubelet/pods/80a7d765-9b80-49f8-8d64-f72ec2acb5d7/volumes" Sep 18 06:44:17.532146 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:17.532111 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6"] Sep 18 06:44:17.532671 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:17.532559 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30aa103a-8c8e-4ee2-af67-0938b4224da4" containerName="revoke-keys" Sep 18 06:44:17.532671 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:17.532571 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="30aa103a-8c8e-4ee2-af67-0938b4224da4" containerName="revoke-keys" Sep 18 06:44:17.532671 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:17.532587 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67f1cb84-a6be-4c3d-bc42-0876725c58aa" containerName="istio-proxy" Sep 18 06:44:17.532671 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:17.532593 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="67f1cb84-a6be-4c3d-bc42-0876725c58aa" containerName="istio-proxy" Sep 18 06:44:17.532671 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:17.532600 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67f1cb84-a6be-4c3d-bc42-0876725c58aa" containerName="istio-proxy" Sep 18 06:44:17.532671 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:17.532607 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="67f1cb84-a6be-4c3d-bc42-0876725c58aa" containerName="istio-proxy" Sep 18 06:44:17.532671 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:17.532615 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80a7d765-9b80-49f8-8d64-f72ec2acb5d7" containerName="payload-processing" Sep 18 06:44:17.532671 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:17.532620 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="80a7d765-9b80-49f8-8d64-f72ec2acb5d7" containerName="payload-processing" Sep 18 06:44:17.532932 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:17.532704 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="67f1cb84-a6be-4c3d-bc42-0876725c58aa" containerName="istio-proxy" Sep 18 06:44:17.532932 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:17.532717 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="30aa103a-8c8e-4ee2-af67-0938b4224da4" containerName="revoke-keys" Sep 18 06:44:17.532932 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:17.532723 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="80a7d765-9b80-49f8-8d64-f72ec2acb5d7" containerName="payload-processing" Sep 18 06:44:17.535993 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:17.535969 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6" Sep 18 06:44:17.538916 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:17.538899 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-14ee59-openshift-default-dockercfg-85glc\"" Sep 18 06:44:17.546815 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:17.546781 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6"] Sep 18 06:44:17.612454 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:17.612424 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/505d2a4d-d115-4775-9f00-85b367cebf79-workload-socket\") pod \"e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6\" (UID: \"505d2a4d-d115-4775-9f00-85b367cebf79\") " pod="openshift-ingress/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6" Sep 18 06:44:17.612616 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:17.612470 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/505d2a4d-d115-4775-9f00-85b367cebf79-istio-podinfo\") pod \"e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6\" (UID: \"505d2a4d-d115-4775-9f00-85b367cebf79\") " pod="openshift-ingress/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6" Sep 18 06:44:17.612616 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:17.612521 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbwcv\" (UniqueName: \"kubernetes.io/projected/505d2a4d-d115-4775-9f00-85b367cebf79-kube-api-access-sbwcv\") pod \"e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6\" (UID: \"505d2a4d-d115-4775-9f00-85b367cebf79\") " pod="openshift-ingress/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6" Sep 18 06:44:17.612616 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:17.612569 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/505d2a4d-d115-4775-9f00-85b367cebf79-istio-token\") pod \"e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6\" (UID: \"505d2a4d-d115-4775-9f00-85b367cebf79\") " pod="openshift-ingress/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6" Sep 18 06:44:17.612616 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:17.612604 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/505d2a4d-d115-4775-9f00-85b367cebf79-istiod-ca-cert\") pod \"e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6\" (UID: \"505d2a4d-d115-4775-9f00-85b367cebf79\") " pod="openshift-ingress/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6" Sep 18 06:44:17.612777 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:17.612624 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/505d2a4d-d115-4775-9f00-85b367cebf79-istio-envoy\") pod \"e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6\" (UID: \"505d2a4d-d115-4775-9f00-85b367cebf79\") " pod="openshift-ingress/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6" Sep 18 06:44:17.612777 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:17.612691 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/505d2a4d-d115-4775-9f00-85b367cebf79-credential-socket\") pod \"e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6\" (UID: \"505d2a4d-d115-4775-9f00-85b367cebf79\") " pod="openshift-ingress/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6" Sep 18 06:44:17.612777 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:17.612734 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/505d2a4d-d115-4775-9f00-85b367cebf79-workload-certs\") pod \"e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6\" (UID: \"505d2a4d-d115-4775-9f00-85b367cebf79\") " pod="openshift-ingress/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6" Sep 18 06:44:17.612777 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:17.612754 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/505d2a4d-d115-4775-9f00-85b367cebf79-istio-data\") pod \"e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6\" (UID: \"505d2a4d-d115-4775-9f00-85b367cebf79\") " pod="openshift-ingress/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6" Sep 18 06:44:17.714331 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:17.714283 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/505d2a4d-d115-4775-9f00-85b367cebf79-credential-socket\") pod \"e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6\" (UID: \"505d2a4d-d115-4775-9f00-85b367cebf79\") " pod="openshift-ingress/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6" Sep 18 06:44:17.714612 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:17.714587 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/505d2a4d-d115-4775-9f00-85b367cebf79-workload-certs\") pod \"e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6\" (UID: \"505d2a4d-d115-4775-9f00-85b367cebf79\") " pod="openshift-ingress/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6" Sep 18 06:44:17.714748 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:17.714734 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/505d2a4d-d115-4775-9f00-85b367cebf79-istio-data\") pod \"e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6\" (UID: \"505d2a4d-d115-4775-9f00-85b367cebf79\") " pod="openshift-ingress/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6" Sep 18 06:44:17.714900 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:17.714873 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/505d2a4d-d115-4775-9f00-85b367cebf79-credential-socket\") pod \"e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6\" (UID: \"505d2a4d-d115-4775-9f00-85b367cebf79\") " pod="openshift-ingress/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6" Sep 18 06:44:17.715061 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:17.715044 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/505d2a4d-d115-4775-9f00-85b367cebf79-workload-socket\") pod \"e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6\" (UID: \"505d2a4d-d115-4775-9f00-85b367cebf79\") " pod="openshift-ingress/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6" Sep 18 06:44:17.715203 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:17.715180 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/505d2a4d-d115-4775-9f00-85b367cebf79-istio-podinfo\") pod \"e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6\" (UID: \"505d2a4d-d115-4775-9f00-85b367cebf79\") " pod="openshift-ingress/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6" Sep 18 06:44:17.715348 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:17.715220 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbwcv\" (UniqueName: \"kubernetes.io/projected/505d2a4d-d115-4775-9f00-85b367cebf79-kube-api-access-sbwcv\") pod \"e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6\" (UID: \"505d2a4d-d115-4775-9f00-85b367cebf79\") " pod="openshift-ingress/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6" Sep 18 06:44:17.715348 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:17.715272 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/505d2a4d-d115-4775-9f00-85b367cebf79-istio-token\") pod \"e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6\" (UID: \"505d2a4d-d115-4775-9f00-85b367cebf79\") " pod="openshift-ingress/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6" Sep 18 06:44:17.715348 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:17.715301 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/505d2a4d-d115-4775-9f00-85b367cebf79-workload-certs\") pod \"e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6\" (UID: \"505d2a4d-d115-4775-9f00-85b367cebf79\") " pod="openshift-ingress/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6" Sep 18 06:44:17.715348 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:17.715315 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/505d2a4d-d115-4775-9f00-85b367cebf79-istio-data\") pod \"e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6\" (UID: \"505d2a4d-d115-4775-9f00-85b367cebf79\") " pod="openshift-ingress/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6" Sep 18 06:44:17.715348 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:17.715327 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/505d2a4d-d115-4775-9f00-85b367cebf79-istiod-ca-cert\") pod \"e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6\" (UID: \"505d2a4d-d115-4775-9f00-85b367cebf79\") " pod="openshift-ingress/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6" Sep 18 06:44:17.715679 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:17.715402 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/505d2a4d-d115-4775-9f00-85b367cebf79-istio-envoy\") pod \"e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6\" (UID: \"505d2a4d-d115-4775-9f00-85b367cebf79\") " pod="openshift-ingress/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6" Sep 18 06:44:17.715679 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:17.715427 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/505d2a4d-d115-4775-9f00-85b367cebf79-workload-socket\") pod \"e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6\" (UID: \"505d2a4d-d115-4775-9f00-85b367cebf79\") " pod="openshift-ingress/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6" Sep 18 06:44:17.716104 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:17.716017 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/505d2a4d-d115-4775-9f00-85b367cebf79-istiod-ca-cert\") pod \"e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6\" (UID: \"505d2a4d-d115-4775-9f00-85b367cebf79\") " pod="openshift-ingress/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6" Sep 18 06:44:17.717795 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:17.717768 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/505d2a4d-d115-4775-9f00-85b367cebf79-istio-envoy\") pod \"e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6\" (UID: \"505d2a4d-d115-4775-9f00-85b367cebf79\") " pod="openshift-ingress/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6" Sep 18 06:44:17.717976 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:17.717954 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/505d2a4d-d115-4775-9f00-85b367cebf79-istio-podinfo\") pod \"e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6\" (UID: \"505d2a4d-d115-4775-9f00-85b367cebf79\") " pod="openshift-ingress/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6" Sep 18 06:44:17.735391 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:17.732467 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/505d2a4d-d115-4775-9f00-85b367cebf79-istio-token\") pod \"e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6\" (UID: \"505d2a4d-d115-4775-9f00-85b367cebf79\") " pod="openshift-ingress/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6" Sep 18 06:44:17.735391 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:17.733535 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbwcv\" (UniqueName: \"kubernetes.io/projected/505d2a4d-d115-4775-9f00-85b367cebf79-kube-api-access-sbwcv\") pod \"e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6\" (UID: \"505d2a4d-d115-4775-9f00-85b367cebf79\") " pod="openshift-ingress/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6" Sep 18 06:44:17.848433 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:17.848399 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6" Sep 18 06:44:18.134403 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:18.128705 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6"] Sep 18 06:44:18.136256 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:18.136218 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:44:18.136430 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:18.136303 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:44:18.136430 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:18.136341 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:44:18.879873 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:18.879834 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6" event={"ID":"505d2a4d-d115-4775-9f00-85b367cebf79","Type":"ContainerStarted","Data":"af993aee090be7459197b964646142f2c590557a330acc6e01edb110de2d5e1d"} Sep 18 06:44:18.880243 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:18.879896 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6" event={"ID":"505d2a4d-d115-4775-9f00-85b367cebf79","Type":"ContainerStarted","Data":"f642bab8483fb2833032fe3584c3ef936ef3af45cb3c64cd36a7cd28d81df912"} Sep 18 06:44:18.900962 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:18.900913 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6" podStartSLOduration=1.900898292 podStartE2EDuration="1.900898292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:44:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:44:18.897839908 +0000 UTC m=+1320.637256104" watchObservedRunningTime="2026-09-18 06:44:18.900898292 +0000 UTC m=+1320.640314465" Sep 18 06:44:19.849045 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:19.849012 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6" Sep 18 06:44:19.853968 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:19.853947 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6" Sep 18 06:44:19.886736 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:19.886709 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6" Sep 18 06:44:19.887549 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:19.887532 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6" Sep 18 06:44:24.337271 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:24.337236 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-14ee59-7b9c6754db-58xcm"] Sep 18 06:44:24.337778 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:24.337764 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="67f1cb84-a6be-4c3d-bc42-0876725c58aa" containerName="istio-proxy" Sep 18 06:44:24.342334 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:24.342309 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-14ee59-7b9c6754db-58xcm" Sep 18 06:44:24.345006 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:24.344976 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-14ee59\"" Sep 18 06:44:24.345158 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:24.345024 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-14ee59-dockercfg-w47rn\"" Sep 18 06:44:24.349437 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:24.349413 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-14ee59-7b9c6754db-58xcm"] Sep 18 06:44:24.380340 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:24.380315 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-14ee59-7784b6b66f-2bcg9"] Sep 18 06:44:24.383769 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:24.383753 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-14ee59-7784b6b66f-2bcg9" Sep 18 06:44:24.392757 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:24.392737 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-14ee59-7784b6b66f-2bcg9"] Sep 18 06:44:24.481208 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:24.481171 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a81a0ebf-f60e-4995-ba97-cfa65c37e81b-plugins-config-volume\") pod \"payload-processing-e2e-explicit-14ee59-7784b6b66f-2bcg9\" (UID: \"a81a0ebf-f60e-4995-ba97-cfa65c37e81b\") " pod="openshift-ingress/payload-processing-e2e-explicit-14ee59-7784b6b66f-2bcg9" Sep 18 06:44:24.481409 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:24.481230 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66c9823b-6600-467a-b1c2-20800910a671-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-14ee59-7b9c6754db-58xcm\" (UID: \"66c9823b-6600-467a-b1c2-20800910a671\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-14ee59-7b9c6754db-58xcm" Sep 18 06:44:24.481409 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:24.481255 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bncdn\" (UniqueName: \"kubernetes.io/projected/66c9823b-6600-467a-b1c2-20800910a671-kube-api-access-bncdn\") pod \"payload-pre-processing-e2e-explicit-14ee59-7b9c6754db-58xcm\" (UID: \"66c9823b-6600-467a-b1c2-20800910a671\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-14ee59-7b9c6754db-58xcm" Sep 18 06:44:24.481409 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:24.481309 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lmrk\" (UniqueName: \"kubernetes.io/projected/a81a0ebf-f60e-4995-ba97-cfa65c37e81b-kube-api-access-5lmrk\") pod \"payload-processing-e2e-explicit-14ee59-7784b6b66f-2bcg9\" (UID: \"a81a0ebf-f60e-4995-ba97-cfa65c37e81b\") " pod="openshift-ingress/payload-processing-e2e-explicit-14ee59-7784b6b66f-2bcg9" Sep 18 06:44:24.582220 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:24.582190 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a81a0ebf-f60e-4995-ba97-cfa65c37e81b-plugins-config-volume\") pod \"payload-processing-e2e-explicit-14ee59-7784b6b66f-2bcg9\" (UID: \"a81a0ebf-f60e-4995-ba97-cfa65c37e81b\") " pod="openshift-ingress/payload-processing-e2e-explicit-14ee59-7784b6b66f-2bcg9" Sep 18 06:44:24.582449 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:24.582232 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66c9823b-6600-467a-b1c2-20800910a671-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-14ee59-7b9c6754db-58xcm\" (UID: \"66c9823b-6600-467a-b1c2-20800910a671\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-14ee59-7b9c6754db-58xcm" Sep 18 06:44:24.582449 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:24.582249 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bncdn\" (UniqueName: \"kubernetes.io/projected/66c9823b-6600-467a-b1c2-20800910a671-kube-api-access-bncdn\") pod \"payload-pre-processing-e2e-explicit-14ee59-7b9c6754db-58xcm\" (UID: \"66c9823b-6600-467a-b1c2-20800910a671\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-14ee59-7b9c6754db-58xcm" Sep 18 06:44:24.582449 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:24.582286 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lmrk\" (UniqueName: \"kubernetes.io/projected/a81a0ebf-f60e-4995-ba97-cfa65c37e81b-kube-api-access-5lmrk\") pod \"payload-processing-e2e-explicit-14ee59-7784b6b66f-2bcg9\" (UID: \"a81a0ebf-f60e-4995-ba97-cfa65c37e81b\") " pod="openshift-ingress/payload-processing-e2e-explicit-14ee59-7784b6b66f-2bcg9" Sep 18 06:44:24.582971 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:24.582948 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a81a0ebf-f60e-4995-ba97-cfa65c37e81b-plugins-config-volume\") pod \"payload-processing-e2e-explicit-14ee59-7784b6b66f-2bcg9\" (UID: \"a81a0ebf-f60e-4995-ba97-cfa65c37e81b\") " pod="openshift-ingress/payload-processing-e2e-explicit-14ee59-7784b6b66f-2bcg9" Sep 18 06:44:24.583059 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:24.582995 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66c9823b-6600-467a-b1c2-20800910a671-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-14ee59-7b9c6754db-58xcm\" (UID: \"66c9823b-6600-467a-b1c2-20800910a671\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-14ee59-7b9c6754db-58xcm" Sep 18 06:44:24.589931 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:24.589879 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lmrk\" (UniqueName: \"kubernetes.io/projected/a81a0ebf-f60e-4995-ba97-cfa65c37e81b-kube-api-access-5lmrk\") pod \"payload-processing-e2e-explicit-14ee59-7784b6b66f-2bcg9\" (UID: \"a81a0ebf-f60e-4995-ba97-cfa65c37e81b\") " pod="openshift-ingress/payload-processing-e2e-explicit-14ee59-7784b6b66f-2bcg9" Sep 18 06:44:24.590076 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:24.590060 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bncdn\" (UniqueName: \"kubernetes.io/projected/66c9823b-6600-467a-b1c2-20800910a671-kube-api-access-bncdn\") pod \"payload-pre-processing-e2e-explicit-14ee59-7b9c6754db-58xcm\" (UID: \"66c9823b-6600-467a-b1c2-20800910a671\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-14ee59-7b9c6754db-58xcm" Sep 18 06:44:24.655041 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:24.655019 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-14ee59-7b9c6754db-58xcm" Sep 18 06:44:24.694529 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:24.694501 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-14ee59-7784b6b66f-2bcg9" Sep 18 06:44:24.927952 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:24.927926 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-14ee59-7b9c6754db-58xcm"] Sep 18 06:44:24.928554 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:44:24.928527 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66c9823b_6600_467a_b1c2_20800910a671.slice/crio-1fa8ee41c7fa9df098fed817efc867e2e461dd301c0bd3afd5813099c2db0e37 WatchSource:0}: Error finding container 1fa8ee41c7fa9df098fed817efc867e2e461dd301c0bd3afd5813099c2db0e37: Status 404 returned error can't find the container with id 1fa8ee41c7fa9df098fed817efc867e2e461dd301c0bd3afd5813099c2db0e37 Sep 18 06:44:24.981358 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:24.981329 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-14ee59-7784b6b66f-2bcg9"] Sep 18 06:44:24.981942 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:44:24.981921 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda81a0ebf_f60e_4995_ba97_cfa65c37e81b.slice/crio-4f093d3a2eeb775b32076d2aca66662370fa23c74d11272a491fa056bb5ed1f2 WatchSource:0}: Error finding container 4f093d3a2eeb775b32076d2aca66662370fa23c74d11272a491fa056bb5ed1f2: Status 404 returned error can't find the container with id 4f093d3a2eeb775b32076d2aca66662370fa23c74d11272a491fa056bb5ed1f2 Sep 18 06:44:25.925335 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:25.925299 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-14ee59-7b9c6754db-58xcm" event={"ID":"66c9823b-6600-467a-b1c2-20800910a671","Type":"ContainerStarted","Data":"ef3b72d204d76c813316ecb79077b3a51678a2a915d0438fb72439c2ece4a1e8"} Sep 18 06:44:25.925790 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:25.925408 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-14ee59-7b9c6754db-58xcm" Sep 18 06:44:25.925790 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:25.925434 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-14ee59-7b9c6754db-58xcm" event={"ID":"66c9823b-6600-467a-b1c2-20800910a671","Type":"ContainerStarted","Data":"1fa8ee41c7fa9df098fed817efc867e2e461dd301c0bd3afd5813099c2db0e37"} Sep 18 06:44:25.927075 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:25.927048 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-14ee59-7784b6b66f-2bcg9" event={"ID":"a81a0ebf-f60e-4995-ba97-cfa65c37e81b","Type":"ContainerStarted","Data":"606c3f1f4d9f58b94bc0029110da6a1190d1f93151b30c74fb2b5e49e8ae2a17"} Sep 18 06:44:25.927196 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:25.927099 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-14ee59-7784b6b66f-2bcg9" event={"ID":"a81a0ebf-f60e-4995-ba97-cfa65c37e81b","Type":"ContainerStarted","Data":"4f093d3a2eeb775b32076d2aca66662370fa23c74d11272a491fa056bb5ed1f2"} Sep 18 06:44:25.927196 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:25.927127 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-14ee59-7784b6b66f-2bcg9" Sep 18 06:44:25.945273 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:25.945202 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-14ee59-7b9c6754db-58xcm" podStartSLOduration=1.945187945 podStartE2EDuration="1.945187945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:44:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:44:25.942260526 +0000 UTC m=+1327.681676722" watchObservedRunningTime="2026-09-18 06:44:25.945187945 +0000 UTC m=+1327.684604118" Sep 18 06:44:25.960730 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:25.960686 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-14ee59-7784b6b66f-2bcg9" podStartSLOduration=1.9606721729999999 podStartE2EDuration="1.960672173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:44:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:44:25.95783567 +0000 UTC m=+1327.697251870" watchObservedRunningTime="2026-09-18 06:44:25.960672173 +0000 UTC m=+1327.700088346" Sep 18 06:44:36.935177 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:36.935088 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-14ee59-7b9c6754db-58xcm" Sep 18 06:44:36.935550 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:36.935312 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-14ee59-7784b6b66f-2bcg9" Sep 18 06:44:48.064930 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:48.064899 2567 generic.go:356] "Generic (PLEG): container finished" podID="505d2a4d-d115-4775-9f00-85b367cebf79" containerID="af993aee090be7459197b964646142f2c590557a330acc6e01edb110de2d5e1d" exitCode=0 Sep 18 06:44:48.065403 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:48.064973 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6" event={"ID":"505d2a4d-d115-4775-9f00-85b367cebf79","Type":"ContainerDied","Data":"af993aee090be7459197b964646142f2c590557a330acc6e01edb110de2d5e1d"} Sep 18 06:44:48.065464 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:48.065451 2567 scope.go:117] "RemoveContainer" containerID="af993aee090be7459197b964646142f2c590557a330acc6e01edb110de2d5e1d" Sep 18 06:44:48.066643 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:48.066616 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:44:48.066711 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:48.066682 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:44:48.066755 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:48.066713 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:44:49.077955 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:49.077918 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6" event={"ID":"505d2a4d-d115-4775-9f00-85b367cebf79","Type":"ContainerStarted","Data":"eb732a675c9d0769b704e29d08654af003c5db59d0a6ed55d4209a54ba32ca11"} Sep 18 06:44:49.849620 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:49.849582 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6" Sep 18 06:44:49.850829 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:49.850800 2567 patch_prober.go:28] interesting pod/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Sep 18 06:44:49.851032 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:49.851003 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6" podUID="505d2a4d-d115-4775-9f00-85b367cebf79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Sep 18 06:44:49.886704 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:49.886671 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6" Sep 18 06:44:50.849353 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:50.849317 2567 patch_prober.go:28] interesting pod/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Sep 18 06:44:50.849973 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:50.849938 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6" podUID="505d2a4d-d115-4775-9f00-85b367cebf79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Sep 18 06:44:51.849337 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:51.849298 2567 patch_prober.go:28] interesting pod/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Sep 18 06:44:51.850425 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:51.850389 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6" podUID="505d2a4d-d115-4775-9f00-85b367cebf79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Sep 18 06:44:52.849321 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:52.849284 2567 patch_prober.go:28] interesting pod/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Sep 18 06:44:52.849667 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:52.849632 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6" podUID="505d2a4d-d115-4775-9f00-85b367cebf79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Sep 18 06:44:53.859390 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:53.857700 2567 patch_prober.go:28] interesting pod/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Sep 18 06:44:53.859390 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:53.857760 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6" podUID="505d2a4d-d115-4775-9f00-85b367cebf79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Sep 18 06:44:54.367394 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:54.366608 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr"] Sep 18 06:44:54.373936 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:54.373914 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr" Sep 18 06:44:54.374335 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:54.374305 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr"] Sep 18 06:44:54.380686 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:54.380663 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-d887938f-openshift-default-dockercfg-ct48n\"" Sep 18 06:44:54.484143 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:54.484103 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8650b750-4f90-46c3-9a28-52c3541d163d-istio-data\") pod \"e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr\" (UID: \"8650b750-4f90-46c3-9a28-52c3541d163d\") " pod="openshift-ingress/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr" Sep 18 06:44:54.484358 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:54.484159 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8650b750-4f90-46c3-9a28-52c3541d163d-istio-podinfo\") pod \"e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr\" (UID: \"8650b750-4f90-46c3-9a28-52c3541d163d\") " pod="openshift-ingress/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr" Sep 18 06:44:54.484358 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:54.484195 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8650b750-4f90-46c3-9a28-52c3541d163d-workload-socket\") pod \"e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr\" (UID: \"8650b750-4f90-46c3-9a28-52c3541d163d\") " pod="openshift-ingress/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr" Sep 18 06:44:54.484358 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:54.484225 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl6qv\" (UniqueName: \"kubernetes.io/projected/8650b750-4f90-46c3-9a28-52c3541d163d-kube-api-access-hl6qv\") pod \"e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr\" (UID: \"8650b750-4f90-46c3-9a28-52c3541d163d\") " pod="openshift-ingress/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr" Sep 18 06:44:54.484358 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:54.484268 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8650b750-4f90-46c3-9a28-52c3541d163d-istio-token\") pod \"e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr\" (UID: \"8650b750-4f90-46c3-9a28-52c3541d163d\") " pod="openshift-ingress/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr" Sep 18 06:44:54.484358 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:54.484319 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8650b750-4f90-46c3-9a28-52c3541d163d-istio-envoy\") pod \"e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr\" (UID: \"8650b750-4f90-46c3-9a28-52c3541d163d\") " pod="openshift-ingress/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr" Sep 18 06:44:54.484843 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:54.484812 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8650b750-4f90-46c3-9a28-52c3541d163d-credential-socket\") pod \"e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr\" (UID: \"8650b750-4f90-46c3-9a28-52c3541d163d\") " pod="openshift-ingress/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr" Sep 18 06:44:54.484965 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:54.484859 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8650b750-4f90-46c3-9a28-52c3541d163d-workload-certs\") pod \"e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr\" (UID: \"8650b750-4f90-46c3-9a28-52c3541d163d\") " pod="openshift-ingress/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr" Sep 18 06:44:54.484965 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:54.484885 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8650b750-4f90-46c3-9a28-52c3541d163d-istiod-ca-cert\") pod \"e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr\" (UID: \"8650b750-4f90-46c3-9a28-52c3541d163d\") " pod="openshift-ingress/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr" Sep 18 06:44:54.585897 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:54.585860 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8650b750-4f90-46c3-9a28-52c3541d163d-credential-socket\") pod \"e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr\" (UID: \"8650b750-4f90-46c3-9a28-52c3541d163d\") " pod="openshift-ingress/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr" Sep 18 06:44:54.586106 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:54.585916 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8650b750-4f90-46c3-9a28-52c3541d163d-workload-certs\") pod \"e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr\" (UID: \"8650b750-4f90-46c3-9a28-52c3541d163d\") " pod="openshift-ingress/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr" Sep 18 06:44:54.586106 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:54.585945 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8650b750-4f90-46c3-9a28-52c3541d163d-istiod-ca-cert\") pod \"e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr\" (UID: \"8650b750-4f90-46c3-9a28-52c3541d163d\") " pod="openshift-ingress/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr" Sep 18 06:44:54.586106 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:54.586058 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8650b750-4f90-46c3-9a28-52c3541d163d-istio-data\") pod \"e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr\" (UID: \"8650b750-4f90-46c3-9a28-52c3541d163d\") " pod="openshift-ingress/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr" Sep 18 06:44:54.586106 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:54.586089 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8650b750-4f90-46c3-9a28-52c3541d163d-istio-podinfo\") pod \"e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr\" (UID: \"8650b750-4f90-46c3-9a28-52c3541d163d\") " pod="openshift-ingress/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr" Sep 18 06:44:54.586465 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:54.586120 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8650b750-4f90-46c3-9a28-52c3541d163d-workload-socket\") pod \"e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr\" (UID: \"8650b750-4f90-46c3-9a28-52c3541d163d\") " pod="openshift-ingress/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr" Sep 18 06:44:54.586465 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:54.586184 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl6qv\" (UniqueName: \"kubernetes.io/projected/8650b750-4f90-46c3-9a28-52c3541d163d-kube-api-access-hl6qv\") pod \"e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr\" (UID: \"8650b750-4f90-46c3-9a28-52c3541d163d\") " pod="openshift-ingress/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr" Sep 18 06:44:54.586465 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:54.586224 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8650b750-4f90-46c3-9a28-52c3541d163d-istio-token\") pod \"e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr\" (UID: \"8650b750-4f90-46c3-9a28-52c3541d163d\") " pod="openshift-ingress/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr" Sep 18 06:44:54.586465 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:54.586301 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8650b750-4f90-46c3-9a28-52c3541d163d-istio-envoy\") pod \"e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr\" (UID: \"8650b750-4f90-46c3-9a28-52c3541d163d\") " pod="openshift-ingress/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr" Sep 18 06:44:54.586465 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:54.586459 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8650b750-4f90-46c3-9a28-52c3541d163d-istio-data\") pod \"e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr\" (UID: \"8650b750-4f90-46c3-9a28-52c3541d163d\") " pod="openshift-ingress/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr" Sep 18 06:44:54.586740 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:54.586498 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8650b750-4f90-46c3-9a28-52c3541d163d-workload-certs\") pod \"e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr\" (UID: \"8650b750-4f90-46c3-9a28-52c3541d163d\") " pod="openshift-ingress/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr" Sep 18 06:44:54.586740 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:54.586534 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8650b750-4f90-46c3-9a28-52c3541d163d-workload-socket\") pod \"e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr\" (UID: \"8650b750-4f90-46c3-9a28-52c3541d163d\") " pod="openshift-ingress/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr" Sep 18 06:44:54.587029 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:54.587003 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8650b750-4f90-46c3-9a28-52c3541d163d-credential-socket\") pod \"e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr\" (UID: \"8650b750-4f90-46c3-9a28-52c3541d163d\") " pod="openshift-ingress/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr" Sep 18 06:44:54.587148 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:54.587099 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8650b750-4f90-46c3-9a28-52c3541d163d-istiod-ca-cert\") pod \"e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr\" (UID: \"8650b750-4f90-46c3-9a28-52c3541d163d\") " pod="openshift-ingress/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr" Sep 18 06:44:54.588957 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:54.588939 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8650b750-4f90-46c3-9a28-52c3541d163d-istio-podinfo\") pod \"e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr\" (UID: \"8650b750-4f90-46c3-9a28-52c3541d163d\") " pod="openshift-ingress/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr" Sep 18 06:44:54.589163 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:54.589144 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8650b750-4f90-46c3-9a28-52c3541d163d-istio-envoy\") pod \"e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr\" (UID: \"8650b750-4f90-46c3-9a28-52c3541d163d\") " pod="openshift-ingress/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr" Sep 18 06:44:54.593879 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:54.593859 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl6qv\" (UniqueName: \"kubernetes.io/projected/8650b750-4f90-46c3-9a28-52c3541d163d-kube-api-access-hl6qv\") pod \"e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr\" (UID: \"8650b750-4f90-46c3-9a28-52c3541d163d\") " pod="openshift-ingress/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr" Sep 18 06:44:54.594117 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:54.594086 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8650b750-4f90-46c3-9a28-52c3541d163d-istio-token\") pod \"e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr\" (UID: \"8650b750-4f90-46c3-9a28-52c3541d163d\") " pod="openshift-ingress/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr" Sep 18 06:44:54.694800 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:54.694711 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr" Sep 18 06:44:54.853940 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:54.853910 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6" Sep 18 06:44:54.854860 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:54.854842 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6" Sep 18 06:44:54.985815 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:54.985786 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr"] Sep 18 06:44:54.986453 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:44:54.986429 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8650b750_4f90_46c3_9a28_52c3541d163d.slice/crio-0157c31a972b62f610a38916de6eaa7b95803f1ef72aa99fbebcab3b4262cf9e WatchSource:0}: Error finding container 0157c31a972b62f610a38916de6eaa7b95803f1ef72aa99fbebcab3b4262cf9e: Status 404 returned error can't find the container with id 0157c31a972b62f610a38916de6eaa7b95803f1ef72aa99fbebcab3b4262cf9e Sep 18 06:44:54.988791 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:54.988759 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:44:54.988877 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:54.988819 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:44:54.988877 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:54.988848 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:44:55.116914 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:55.116874 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr" event={"ID":"8650b750-4f90-46c3-9a28-52c3541d163d","Type":"ContainerStarted","Data":"b82857e607e1666c0737abf89991dd370550780720115c6a5088d33a1badfb07"} Sep 18 06:44:55.117097 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:55.116932 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr" event={"ID":"8650b750-4f90-46c3-9a28-52c3541d163d","Type":"ContainerStarted","Data":"0157c31a972b62f610a38916de6eaa7b95803f1ef72aa99fbebcab3b4262cf9e"} Sep 18 06:44:55.160106 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:55.160045 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr" podStartSLOduration=1.16002386 podStartE2EDuration="1.16002386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:44:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:44:55.155071656 +0000 UTC m=+1356.894487860" watchObservedRunningTime="2026-09-18 06:44:55.16002386 +0000 UTC m=+1356.899440038" Sep 18 06:44:55.695300 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:55.695267 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr" Sep 18 06:44:56.700250 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:56.700215 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr" Sep 18 06:44:57.132715 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:57.132685 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr" Sep 18 06:44:57.133659 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:44:57.133639 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr" Sep 18 06:45:00.145166 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:00.145133 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-654463-29828565-zkvz8"] Sep 18 06:45:00.151123 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:00.151098 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29828565-g2dqc"] Sep 18 06:45:00.151286 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:00.151172 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-654463-29828565-zkvz8" Sep 18 06:45:00.153755 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:00.153731 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-88t7h\"" Sep 18 06:45:00.154604 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:00.154583 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-14ee59-29828565-l7l28"] Sep 18 06:45:00.155652 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:00.154812 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828565-g2dqc" Sep 18 06:45:00.159029 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:00.159007 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-708b6f-29828565-g4m2z"] Sep 18 06:45:00.159949 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:00.159817 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-14ee59-29828565-l7l28" Sep 18 06:45:00.166554 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:00.166534 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-654463-29828565-zkvz8"] Sep 18 06:45:00.166741 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:00.166714 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-14ee59-29828565-l7l28"] Sep 18 06:45:00.166995 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:00.166981 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29828565-g2dqc"] Sep 18 06:45:00.167142 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:00.166925 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-708b6f-29828565-g4m2z" Sep 18 06:45:00.169066 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:00.169045 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-654463-29828565-zkvz8" Sep 18 06:45:00.171011 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:00.170977 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-708b6f-29828565-g4m2z"] Sep 18 06:45:00.179425 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:00.179390 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828565-g2dqc" Sep 18 06:45:00.212829 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:00.212800 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-14ee59-29828565-l7l28" Sep 18 06:45:00.221560 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:00.221537 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-708b6f-29828565-g4m2z" Sep 18 06:45:00.460879 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:00.460588 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-654463-29828565-zkvz8"] Sep 18 06:45:00.469132 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:00.469107 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29828565-g2dqc"] Sep 18 06:45:00.469995 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:45:00.469971 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c07b0b7_0e00_44e1_b6f2_9b50968d0d8b.slice/crio-57b33cfe161ba6f8a20b7ad55ea87c060e108e7f5da23378c05dafbc14442990 WatchSource:0}: Error finding container 57b33cfe161ba6f8a20b7ad55ea87c060e108e7f5da23378c05dafbc14442990: Status 404 returned error can't find the container with id 57b33cfe161ba6f8a20b7ad55ea87c060e108e7f5da23378c05dafbc14442990 Sep 18 06:45:00.509299 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:00.509274 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-14ee59-29828565-l7l28"] Sep 18 06:45:00.509958 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:45:00.509937 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe813339_652d_4bff_8403_415a0346ae0e.slice/crio-75b12911ce8fc9186edcf001202c0b548ecb776b9f31a6100fcc084d0f473381 WatchSource:0}: Error finding container 75b12911ce8fc9186edcf001202c0b548ecb776b9f31a6100fcc084d0f473381: Status 404 returned error can't find the container with id 75b12911ce8fc9186edcf001202c0b548ecb776b9f31a6100fcc084d0f473381 Sep 18 06:45:00.534705 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:00.534683 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-708b6f-29828565-g4m2z"] Sep 18 06:45:00.543538 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:45:00.543515 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod263178e6_66a0_4b7c_8b94_d4337e921084.slice/crio-5c84233f0d04ab6ffee1e1c3641e6311715e5e60c4ab010931794ac70ead7239 WatchSource:0}: Error finding container 5c84233f0d04ab6ffee1e1c3641e6311715e5e60c4ab010931794ac70ead7239: Status 404 returned error can't find the container with id 5c84233f0d04ab6ffee1e1c3641e6311715e5e60c4ab010931794ac70ead7239 Sep 18 06:45:01.159105 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:01.159070 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828565-g2dqc" event={"ID":"2c07b0b7-0e00-44e1-b6f2-9b50968d0d8b","Type":"ContainerStarted","Data":"05078d0890b69a3599913a322834405b908052c82f2904630134cbc9d9159a39"} Sep 18 06:45:01.159506 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:01.159130 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828565-g2dqc" event={"ID":"2c07b0b7-0e00-44e1-b6f2-9b50968d0d8b","Type":"ContainerStarted","Data":"57b33cfe161ba6f8a20b7ad55ea87c060e108e7f5da23378c05dafbc14442990"} Sep 18 06:45:01.160741 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:01.160704 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-654463-29828565-zkvz8" event={"ID":"1dadf7d7-cd98-4e5b-b478-98546ac71a29","Type":"ContainerStarted","Data":"244e248746694d6da7a9857870cc225dd90d7fb9545dc6fefa63204c31021aa1"} Sep 18 06:45:01.160886 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:01.160751 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-654463-29828565-zkvz8" event={"ID":"1dadf7d7-cd98-4e5b-b478-98546ac71a29","Type":"ContainerStarted","Data":"898452af0cadc498dbea239434e1f1ee045c632d6a899efef03df7b6e71ee22e"} Sep 18 06:45:01.162254 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:01.162228 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-708b6f-29828565-g4m2z" event={"ID":"263178e6-66a0-4b7c-8b94-d4337e921084","Type":"ContainerStarted","Data":"48994365af992addd259b184a572e9333033b45b9c17e49f5694b8388bab224c"} Sep 18 06:45:01.162385 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:01.162278 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-708b6f-29828565-g4m2z" event={"ID":"263178e6-66a0-4b7c-8b94-d4337e921084","Type":"ContainerStarted","Data":"5c84233f0d04ab6ffee1e1c3641e6311715e5e60c4ab010931794ac70ead7239"} Sep 18 06:45:01.163739 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:01.163718 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-14ee59-29828565-l7l28" event={"ID":"be813339-652d-4bff-8403-415a0346ae0e","Type":"ContainerStarted","Data":"a30c8454b663410e7d2c73da02097705596769b015bb51bfda72aa9441b7e766"} Sep 18 06:45:01.163860 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:01.163765 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-14ee59-29828565-l7l28" event={"ID":"be813339-652d-4bff-8403-415a0346ae0e","Type":"ContainerStarted","Data":"75b12911ce8fc9186edcf001202c0b548ecb776b9f31a6100fcc084d0f473381"} Sep 18 06:45:01.173519 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:01.173477 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828565-g2dqc" podStartSLOduration=1.173463267 podStartE2EDuration="1.173463267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:45:01.173045164 +0000 UTC m=+1362.912461377" watchObservedRunningTime="2026-09-18 06:45:01.173463267 +0000 UTC m=+1362.912879441" Sep 18 06:45:01.187035 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:01.186995 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-708b6f-29828565-g4m2z" podStartSLOduration=1.186979448 podStartE2EDuration="1.186979448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:45:01.186792116 +0000 UTC m=+1362.926208307" watchObservedRunningTime="2026-09-18 06:45:01.186979448 +0000 UTC m=+1362.926395621" Sep 18 06:45:01.205449 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:01.205408 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-654463-29828565-zkvz8" podStartSLOduration=1.205394243 podStartE2EDuration="1.205394243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:45:01.202816145 +0000 UTC m=+1362.942232341" watchObservedRunningTime="2026-09-18 06:45:01.205394243 +0000 UTC m=+1362.944810418" Sep 18 06:45:01.218215 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:01.218172 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-14ee59-29828565-l7l28" podStartSLOduration=1.218161077 podStartE2EDuration="1.218161077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:45:01.21532719 +0000 UTC m=+1362.954743388" watchObservedRunningTime="2026-09-18 06:45:01.218161077 +0000 UTC m=+1362.957577313" Sep 18 06:45:02.172634 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:02.172599 2567 generic.go:356] "Generic (PLEG): container finished" podID="be813339-652d-4bff-8403-415a0346ae0e" containerID="a30c8454b663410e7d2c73da02097705596769b015bb51bfda72aa9441b7e766" exitCode=0 Sep 18 06:45:02.173018 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:02.172680 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-14ee59-29828565-l7l28" event={"ID":"be813339-652d-4bff-8403-415a0346ae0e","Type":"ContainerDied","Data":"a30c8454b663410e7d2c73da02097705596769b015bb51bfda72aa9441b7e766"} Sep 18 06:45:02.174457 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:02.174432 2567 generic.go:356] "Generic (PLEG): container finished" podID="2c07b0b7-0e00-44e1-b6f2-9b50968d0d8b" containerID="05078d0890b69a3599913a322834405b908052c82f2904630134cbc9d9159a39" exitCode=0 Sep 18 06:45:02.174567 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:02.174499 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828565-g2dqc" event={"ID":"2c07b0b7-0e00-44e1-b6f2-9b50968d0d8b","Type":"ContainerDied","Data":"05078d0890b69a3599913a322834405b908052c82f2904630134cbc9d9159a39"} Sep 18 06:45:02.176066 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:02.176043 2567 generic.go:356] "Generic (PLEG): container finished" podID="1dadf7d7-cd98-4e5b-b478-98546ac71a29" containerID="244e248746694d6da7a9857870cc225dd90d7fb9545dc6fefa63204c31021aa1" exitCode=0 Sep 18 06:45:02.176201 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:02.176125 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-654463-29828565-zkvz8" event={"ID":"1dadf7d7-cd98-4e5b-b478-98546ac71a29","Type":"ContainerDied","Data":"244e248746694d6da7a9857870cc225dd90d7fb9545dc6fefa63204c31021aa1"} Sep 18 06:45:02.177760 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:02.177738 2567 generic.go:356] "Generic (PLEG): container finished" podID="263178e6-66a0-4b7c-8b94-d4337e921084" containerID="48994365af992addd259b184a572e9333033b45b9c17e49f5694b8388bab224c" exitCode=0 Sep 18 06:45:02.177875 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:02.177761 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-708b6f-29828565-g4m2z" event={"ID":"263178e6-66a0-4b7c-8b94-d4337e921084","Type":"ContainerDied","Data":"48994365af992addd259b184a572e9333033b45b9c17e49f5694b8388bab224c"} Sep 18 06:45:03.324478 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:03.324459 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-708b6f-29828565-g4m2z" Sep 18 06:45:03.358816 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:03.358786 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828565-g2dqc" Sep 18 06:45:03.420339 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:03.420315 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-654463-29828565-zkvz8" Sep 18 06:45:03.423802 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:03.423782 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-14ee59-29828565-l7l28" Sep 18 06:45:04.032324 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:04.032289 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d887938f-5645b797b6-q8vkj"] Sep 18 06:45:04.032810 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:04.032796 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="263178e6-66a0-4b7c-8b94-d4337e921084" containerName="cleanup" Sep 18 06:45:04.032853 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:04.032813 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="263178e6-66a0-4b7c-8b94-d4337e921084" containerName="cleanup" Sep 18 06:45:04.032853 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:04.032824 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dadf7d7-cd98-4e5b-b478-98546ac71a29" containerName="cleanup" Sep 18 06:45:04.032853 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:04.032830 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dadf7d7-cd98-4e5b-b478-98546ac71a29" containerName="cleanup" Sep 18 06:45:04.032853 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:04.032843 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be813339-652d-4bff-8403-415a0346ae0e" containerName="cleanup" Sep 18 06:45:04.032853 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:04.032849 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="be813339-652d-4bff-8403-415a0346ae0e" containerName="cleanup" Sep 18 06:45:04.033006 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:04.032856 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c07b0b7-0e00-44e1-b6f2-9b50968d0d8b" containerName="cleanup" Sep 18 06:45:04.033006 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:04.032861 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c07b0b7-0e00-44e1-b6f2-9b50968d0d8b" containerName="cleanup" Sep 18 06:45:04.033006 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:04.032933 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c07b0b7-0e00-44e1-b6f2-9b50968d0d8b" containerName="cleanup" Sep 18 06:45:04.033006 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:04.032941 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="be813339-652d-4bff-8403-415a0346ae0e" containerName="cleanup" Sep 18 06:45:04.033006 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:04.032951 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="263178e6-66a0-4b7c-8b94-d4337e921084" containerName="cleanup" Sep 18 06:45:04.033006 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:04.032958 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dadf7d7-cd98-4e5b-b478-98546ac71a29" containerName="cleanup" Sep 18 06:45:04.036154 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:04.036135 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d887938f-5645b797b6-q8vkj" Sep 18 06:45:04.038672 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:04.038651 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-d887938f\"" Sep 18 06:45:04.038767 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:04.038688 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-d887938f-dockercfg-55pl8\"" Sep 18 06:45:04.047442 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:04.047418 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d887938f-5645b797b6-q8vkj"] Sep 18 06:45:04.175885 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:04.175854 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7ngn\" (UniqueName: \"kubernetes.io/projected/046679b1-2638-42e1-ace9-582e569abdbb-kube-api-access-x7ngn\") pod \"payload-pre-processing-e2e-mt-d887938f-5645b797b6-q8vkj\" (UID: \"046679b1-2638-42e1-ace9-582e569abdbb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d887938f-5645b797b6-q8vkj" Sep 18 06:45:04.175885 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:04.175890 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/046679b1-2638-42e1-ace9-582e569abdbb-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d887938f-5645b797b6-q8vkj\" (UID: \"046679b1-2638-42e1-ace9-582e569abdbb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d887938f-5645b797b6-q8vkj" Sep 18 06:45:04.193895 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:04.193871 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-708b6f-29828565-g4m2z" Sep 18 06:45:04.194038 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:04.193870 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-708b6f-29828565-g4m2z" event={"ID":"263178e6-66a0-4b7c-8b94-d4337e921084","Type":"ContainerDied","Data":"5c84233f0d04ab6ffee1e1c3641e6311715e5e60c4ab010931794ac70ead7239"} Sep 18 06:45:04.194038 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:04.194007 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c84233f0d04ab6ffee1e1c3641e6311715e5e60c4ab010931794ac70ead7239" Sep 18 06:45:04.195359 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:04.195341 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-14ee59-29828565-l7l28" Sep 18 06:45:04.195486 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:04.195397 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-14ee59-29828565-l7l28" event={"ID":"be813339-652d-4bff-8403-415a0346ae0e","Type":"ContainerDied","Data":"75b12911ce8fc9186edcf001202c0b548ecb776b9f31a6100fcc084d0f473381"} Sep 18 06:45:04.195486 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:04.195439 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75b12911ce8fc9186edcf001202c0b548ecb776b9f31a6100fcc084d0f473381" Sep 18 06:45:04.196743 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:04.196728 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828565-g2dqc" Sep 18 06:45:04.196843 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:04.196758 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828565-g2dqc" event={"ID":"2c07b0b7-0e00-44e1-b6f2-9b50968d0d8b","Type":"ContainerDied","Data":"57b33cfe161ba6f8a20b7ad55ea87c060e108e7f5da23378c05dafbc14442990"} Sep 18 06:45:04.196843 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:04.196801 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57b33cfe161ba6f8a20b7ad55ea87c060e108e7f5da23378c05dafbc14442990" Sep 18 06:45:04.198141 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:04.198123 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-654463-29828565-zkvz8" Sep 18 06:45:04.198141 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:04.198133 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-654463-29828565-zkvz8" event={"ID":"1dadf7d7-cd98-4e5b-b478-98546ac71a29","Type":"ContainerDied","Data":"898452af0cadc498dbea239434e1f1ee045c632d6a899efef03df7b6e71ee22e"} Sep 18 06:45:04.198295 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:04.198169 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="898452af0cadc498dbea239434e1f1ee045c632d6a899efef03df7b6e71ee22e" Sep 18 06:45:04.276956 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:04.276929 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7ngn\" (UniqueName: \"kubernetes.io/projected/046679b1-2638-42e1-ace9-582e569abdbb-kube-api-access-x7ngn\") pod \"payload-pre-processing-e2e-mt-d887938f-5645b797b6-q8vkj\" (UID: \"046679b1-2638-42e1-ace9-582e569abdbb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d887938f-5645b797b6-q8vkj" Sep 18 06:45:04.277061 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:04.276964 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/046679b1-2638-42e1-ace9-582e569abdbb-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d887938f-5645b797b6-q8vkj\" (UID: \"046679b1-2638-42e1-ace9-582e569abdbb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d887938f-5645b797b6-q8vkj" Sep 18 06:45:04.277700 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:04.277682 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/046679b1-2638-42e1-ace9-582e569abdbb-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d887938f-5645b797b6-q8vkj\" (UID: \"046679b1-2638-42e1-ace9-582e569abdbb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d887938f-5645b797b6-q8vkj" Sep 18 06:45:04.296128 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:04.296073 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7ngn\" (UniqueName: \"kubernetes.io/projected/046679b1-2638-42e1-ace9-582e569abdbb-kube-api-access-x7ngn\") pod \"payload-pre-processing-e2e-mt-d887938f-5645b797b6-q8vkj\" (UID: \"046679b1-2638-42e1-ace9-582e569abdbb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d887938f-5645b797b6-q8vkj" Sep 18 06:45:04.346773 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:04.346753 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d887938f-5645b797b6-q8vkj" Sep 18 06:45:04.621403 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:04.621375 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d887938f-5645b797b6-q8vkj"] Sep 18 06:45:04.621938 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:45:04.621919 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod046679b1_2638_42e1_ace9_582e569abdbb.slice/crio-33f0c6edc129667ec59655baa881936b49c1ea804324278cb506b2ac6c4cb4fa WatchSource:0}: Error finding container 33f0c6edc129667ec59655baa881936b49c1ea804324278cb506b2ac6c4cb4fa: Status 404 returned error can't find the container with id 33f0c6edc129667ec59655baa881936b49c1ea804324278cb506b2ac6c4cb4fa Sep 18 06:45:05.205997 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:05.205969 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d887938f-5645b797b6-q8vkj" event={"ID":"046679b1-2638-42e1-ace9-582e569abdbb","Type":"ContainerStarted","Data":"ad76219174936ffdef92eb6cace17a69e2286812f93534083196e106d09fcb1c"} Sep 18 06:45:05.206160 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:05.206016 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d887938f-5645b797b6-q8vkj" event={"ID":"046679b1-2638-42e1-ace9-582e569abdbb","Type":"ContainerStarted","Data":"33f0c6edc129667ec59655baa881936b49c1ea804324278cb506b2ac6c4cb4fa"} Sep 18 06:45:05.206160 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:05.206107 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d887938f-5645b797b6-q8vkj" Sep 18 06:45:05.219571 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:05.219527 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-d887938f-5645b797b6-q8vkj" podStartSLOduration=1.219514519 podStartE2EDuration="1.219514519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:45:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:45:05.218643377 +0000 UTC m=+1366.958059571" watchObservedRunningTime="2026-09-18 06:45:05.219514519 +0000 UTC m=+1366.958930693" Sep 18 06:45:16.215221 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:16.215192 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d887938f-5645b797b6-q8vkj" Sep 18 06:45:26.344354 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:26.344321 2567 generic.go:356] "Generic (PLEG): container finished" podID="8650b750-4f90-46c3-9a28-52c3541d163d" containerID="b82857e607e1666c0737abf89991dd370550780720115c6a5088d33a1badfb07" exitCode=0 Sep 18 06:45:26.344755 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:26.344388 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr" event={"ID":"8650b750-4f90-46c3-9a28-52c3541d163d","Type":"ContainerDied","Data":"b82857e607e1666c0737abf89991dd370550780720115c6a5088d33a1badfb07"} Sep 18 06:45:26.344832 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:26.344813 2567 scope.go:117] "RemoveContainer" containerID="b82857e607e1666c0737abf89991dd370550780720115c6a5088d33a1badfb07" Sep 18 06:45:26.345980 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:26.345951 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:45:26.346050 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:26.346037 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:45:26.346092 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:26.346072 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:45:27.133088 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:27.133057 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr" Sep 18 06:45:27.356784 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:27.356742 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr" event={"ID":"8650b750-4f90-46c3-9a28-52c3541d163d","Type":"ContainerStarted","Data":"9cbbaf8fabdfcd119ae228c42e6656eea821039c78aeb26a0980664b5e14a4da"} Sep 18 06:45:27.695745 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:27.695709 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr" Sep 18 06:45:27.697138 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:27.697077 2567 patch_prober.go:28] interesting pod/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Sep 18 06:45:27.697270 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:27.697174 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr" podUID="8650b750-4f90-46c3-9a28-52c3541d163d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Sep 18 06:45:28.696038 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:28.695999 2567 patch_prober.go:28] interesting pod/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Sep 18 06:45:28.696533 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:28.696083 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr" podUID="8650b750-4f90-46c3-9a28-52c3541d163d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Sep 18 06:45:29.695595 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:29.695560 2567 patch_prober.go:28] interesting pod/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Sep 18 06:45:29.695772 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:29.695636 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr" podUID="8650b750-4f90-46c3-9a28-52c3541d163d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Sep 18 06:45:30.695183 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:30.695147 2567 patch_prober.go:28] interesting pod/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Sep 18 06:45:30.695633 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:30.695198 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr" podUID="8650b750-4f90-46c3-9a28-52c3541d163d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Sep 18 06:45:31.695805 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:31.695772 2567 patch_prober.go:28] interesting pod/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Sep 18 06:45:31.696178 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:31.695831 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr" podUID="8650b750-4f90-46c3-9a28-52c3541d163d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Sep 18 06:45:32.699082 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:32.699051 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr" Sep 18 06:45:32.699566 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:32.699545 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr" Sep 18 06:45:32.700277 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:32.700258 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr" Sep 18 06:45:36.851232 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:36.851198 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d887938f-lv5bv"] Sep 18 06:45:36.854392 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:36.854357 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d887938f-lv5bv" Sep 18 06:45:36.859120 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:36.859101 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 18 06:45:36.859260 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:36.859234 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-88t7h\"" Sep 18 06:45:36.872933 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:36.872911 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d887938f-lv5bv"] Sep 18 06:45:36.984285 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:36.984255 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68671084-e485-4b2f-9b88-9ee70253314f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-d887938f-lv5bv\" (UID: \"68671084-e485-4b2f-9b88-9ee70253314f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d887938f-lv5bv" Sep 18 06:45:37.085150 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:37.085118 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68671084-e485-4b2f-9b88-9ee70253314f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-d887938f-lv5bv\" (UID: \"68671084-e485-4b2f-9b88-9ee70253314f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d887938f-lv5bv" Sep 18 06:45:37.085788 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:37.085771 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68671084-e485-4b2f-9b88-9ee70253314f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-d887938f-lv5bv\" (UID: \"68671084-e485-4b2f-9b88-9ee70253314f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d887938f-lv5bv" Sep 18 06:45:37.164613 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:37.164558 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d887938f-lv5bv" Sep 18 06:45:37.434626 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:37.434593 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d887938f-lv5bv"] Sep 18 06:45:37.436043 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:45:37.436017 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68671084_e485_4b2f_9b88_9ee70253314f.slice/crio-b59b860bd914097df3ca944796246d83d6aa441828917de2c73785f73d9620d7 WatchSource:0}: Error finding container b59b860bd914097df3ca944796246d83d6aa441828917de2c73785f73d9620d7: Status 404 returned error can't find the container with id b59b860bd914097df3ca944796246d83d6aa441828917de2c73785f73d9620d7 Sep 18 06:45:38.444602 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:38.444503 2567 generic.go:356] "Generic (PLEG): container finished" podID="68671084-e485-4b2f-9b88-9ee70253314f" containerID="96463803e08debcf3a780e8caa47a5f7d4380455812521a46e9dc6a0079bc6d8" exitCode=0 Sep 18 06:45:38.446591 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:38.446564 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d887938f-lv5bv" event={"ID":"68671084-e485-4b2f-9b88-9ee70253314f","Type":"ContainerDied","Data":"96463803e08debcf3a780e8caa47a5f7d4380455812521a46e9dc6a0079bc6d8"} Sep 18 06:45:38.446699 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:38.446610 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d887938f-lv5bv" event={"ID":"68671084-e485-4b2f-9b88-9ee70253314f","Type":"ContainerStarted","Data":"b59b860bd914097df3ca944796246d83d6aa441828917de2c73785f73d9620d7"} Sep 18 06:45:39.595014 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:39.594991 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d887938f-lv5bv" Sep 18 06:45:39.711118 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:39.711039 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68671084-e485-4b2f-9b88-9ee70253314f-maas-api-service-ca\") pod \"68671084-e485-4b2f-9b88-9ee70253314f\" (UID: \"68671084-e485-4b2f-9b88-9ee70253314f\") " Sep 18 06:45:39.711381 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:39.711335 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68671084-e485-4b2f-9b88-9ee70253314f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "68671084-e485-4b2f-9b88-9ee70253314f" (UID: "68671084-e485-4b2f-9b88-9ee70253314f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 06:45:39.812105 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:39.812053 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68671084-e485-4b2f-9b88-9ee70253314f-maas-api-service-ca\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:45:40.458581 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:40.458555 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d887938f-lv5bv" Sep 18 06:45:40.458581 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:40.458564 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d887938f-lv5bv" event={"ID":"68671084-e485-4b2f-9b88-9ee70253314f","Type":"ContainerDied","Data":"b59b860bd914097df3ca944796246d83d6aa441828917de2c73785f73d9620d7"} Sep 18 06:45:40.458733 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:40.458609 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b59b860bd914097df3ca944796246d83d6aa441828917de2c73785f73d9620d7" Sep 18 06:45:47.382019 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:47.381982 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d887938f-5645b797b6-q8vkj"] Sep 18 06:45:47.382517 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:47.382248 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-d887938f-5645b797b6-q8vkj" podUID="046679b1-2638-42e1-ace9-582e569abdbb" containerName="payload-pre-processing" containerID="cri-o://ad76219174936ffdef92eb6cace17a69e2286812f93534083196e106d09fcb1c" gracePeriod=30 Sep 18 06:45:47.506602 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:47.506569 2567 generic.go:356] "Generic (PLEG): container finished" podID="046679b1-2638-42e1-ace9-582e569abdbb" containerID="ad76219174936ffdef92eb6cace17a69e2286812f93534083196e106d09fcb1c" exitCode=0 Sep 18 06:45:47.506742 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:47.506639 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d887938f-5645b797b6-q8vkj" event={"ID":"046679b1-2638-42e1-ace9-582e569abdbb","Type":"ContainerDied","Data":"ad76219174936ffdef92eb6cace17a69e2286812f93534083196e106d09fcb1c"} Sep 18 06:45:47.605010 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:47.604954 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d887938f-lv5bv"] Sep 18 06:45:47.610429 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:47.610351 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d887938f-lv5bv"] Sep 18 06:45:47.655552 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:47.655526 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d887938f-5645b797b6-q8vkj" Sep 18 06:45:47.788654 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:47.788621 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/046679b1-2638-42e1-ace9-582e569abdbb-plugins-config-volume\") pod \"046679b1-2638-42e1-ace9-582e569abdbb\" (UID: \"046679b1-2638-42e1-ace9-582e569abdbb\") " Sep 18 06:45:47.788845 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:47.788722 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x7ngn\" (UniqueName: \"kubernetes.io/projected/046679b1-2638-42e1-ace9-582e569abdbb-kube-api-access-x7ngn\") pod \"046679b1-2638-42e1-ace9-582e569abdbb\" (UID: \"046679b1-2638-42e1-ace9-582e569abdbb\") " Sep 18 06:45:47.789043 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:47.789017 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/046679b1-2638-42e1-ace9-582e569abdbb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "046679b1-2638-42e1-ace9-582e569abdbb" (UID: "046679b1-2638-42e1-ace9-582e569abdbb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 06:45:47.790835 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:47.790813 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/046679b1-2638-42e1-ace9-582e569abdbb-kube-api-access-x7ngn" (OuterVolumeSpecName: "kube-api-access-x7ngn") pod "046679b1-2638-42e1-ace9-582e569abdbb" (UID: "046679b1-2638-42e1-ace9-582e569abdbb"). InnerVolumeSpecName "kube-api-access-x7ngn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 06:45:47.890054 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:47.889983 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/046679b1-2638-42e1-ace9-582e569abdbb-plugins-config-volume\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:45:47.890054 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:47.890008 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7ngn\" (UniqueName: \"kubernetes.io/projected/046679b1-2638-42e1-ace9-582e569abdbb-kube-api-access-x7ngn\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:45:48.446905 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:48.446869 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68671084-e485-4b2f-9b88-9ee70253314f" path="/var/lib/kubelet/pods/68671084-e485-4b2f-9b88-9ee70253314f/volumes" Sep 18 06:45:48.513519 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:48.513495 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d887938f-5645b797b6-q8vkj" Sep 18 06:45:48.513651 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:48.513542 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d887938f-5645b797b6-q8vkj" event={"ID":"046679b1-2638-42e1-ace9-582e569abdbb","Type":"ContainerDied","Data":"33f0c6edc129667ec59655baa881936b49c1ea804324278cb506b2ac6c4cb4fa"} Sep 18 06:45:48.513651 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:48.513597 2567 scope.go:117] "RemoveContainer" containerID="ad76219174936ffdef92eb6cace17a69e2286812f93534083196e106d09fcb1c" Sep 18 06:45:48.540806 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:48.540777 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d887938f-5645b797b6-q8vkj"] Sep 18 06:45:48.544070 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:48.544043 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d887938f-5645b797b6-q8vkj"] Sep 18 06:45:50.446074 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:50.446042 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="046679b1-2638-42e1-ace9-582e569abdbb" path="/var/lib/kubelet/pods/046679b1-2638-42e1-ace9-582e569abdbb/volumes" Sep 18 06:45:53.883122 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:53.883092 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr"] Sep 18 06:45:53.883668 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:53.883312 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr" podUID="8650b750-4f90-46c3-9a28-52c3541d163d" containerName="istio-proxy" containerID="cri-o://9cbbaf8fabdfcd119ae228c42e6656eea821039c78aeb26a0980664b5e14a4da" gracePeriod=30 Sep 18 06:45:59.147415 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:59.147335 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr" Sep 18 06:45:59.186738 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:59.186709 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8650b750-4f90-46c3-9a28-52c3541d163d-credential-socket\") pod \"8650b750-4f90-46c3-9a28-52c3541d163d\" (UID: \"8650b750-4f90-46c3-9a28-52c3541d163d\") " Sep 18 06:45:59.186872 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:59.186756 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hl6qv\" (UniqueName: \"kubernetes.io/projected/8650b750-4f90-46c3-9a28-52c3541d163d-kube-api-access-hl6qv\") pod \"8650b750-4f90-46c3-9a28-52c3541d163d\" (UID: \"8650b750-4f90-46c3-9a28-52c3541d163d\") " Sep 18 06:45:59.186872 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:59.186854 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8650b750-4f90-46c3-9a28-52c3541d163d-workload-certs\") pod \"8650b750-4f90-46c3-9a28-52c3541d163d\" (UID: \"8650b750-4f90-46c3-9a28-52c3541d163d\") " Sep 18 06:45:59.186998 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:59.186888 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8650b750-4f90-46c3-9a28-52c3541d163d-workload-socket\") pod \"8650b750-4f90-46c3-9a28-52c3541d163d\" (UID: \"8650b750-4f90-46c3-9a28-52c3541d163d\") " Sep 18 06:45:59.186998 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:59.186920 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8650b750-4f90-46c3-9a28-52c3541d163d-istiod-ca-cert\") pod \"8650b750-4f90-46c3-9a28-52c3541d163d\" (UID: \"8650b750-4f90-46c3-9a28-52c3541d163d\") " Sep 18 06:45:59.186998 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:59.186973 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8650b750-4f90-46c3-9a28-52c3541d163d-istio-podinfo\") pod \"8650b750-4f90-46c3-9a28-52c3541d163d\" (UID: \"8650b750-4f90-46c3-9a28-52c3541d163d\") " Sep 18 06:45:59.187152 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:59.186968 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8650b750-4f90-46c3-9a28-52c3541d163d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8650b750-4f90-46c3-9a28-52c3541d163d" (UID: "8650b750-4f90-46c3-9a28-52c3541d163d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:45:59.187152 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:59.187027 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8650b750-4f90-46c3-9a28-52c3541d163d-istio-envoy\") pod \"8650b750-4f90-46c3-9a28-52c3541d163d\" (UID: \"8650b750-4f90-46c3-9a28-52c3541d163d\") " Sep 18 06:45:59.187152 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:59.187110 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8650b750-4f90-46c3-9a28-52c3541d163d-istio-token\") pod \"8650b750-4f90-46c3-9a28-52c3541d163d\" (UID: \"8650b750-4f90-46c3-9a28-52c3541d163d\") " Sep 18 06:45:59.187152 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:59.187143 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8650b750-4f90-46c3-9a28-52c3541d163d-istio-data\") pod \"8650b750-4f90-46c3-9a28-52c3541d163d\" (UID: \"8650b750-4f90-46c3-9a28-52c3541d163d\") " Sep 18 06:45:59.187387 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:59.187291 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8650b750-4f90-46c3-9a28-52c3541d163d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8650b750-4f90-46c3-9a28-52c3541d163d" (UID: "8650b750-4f90-46c3-9a28-52c3541d163d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 06:45:59.187550 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:59.187528 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8650b750-4f90-46c3-9a28-52c3541d163d-credential-socket\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:45:59.187619 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:59.187549 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8650b750-4f90-46c3-9a28-52c3541d163d-istio-data" (OuterVolumeSpecName: "istio-data") pod "8650b750-4f90-46c3-9a28-52c3541d163d" (UID: "8650b750-4f90-46c3-9a28-52c3541d163d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:45:59.187619 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:59.187555 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8650b750-4f90-46c3-9a28-52c3541d163d-istiod-ca-cert\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:45:59.187743 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:59.187719 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8650b750-4f90-46c3-9a28-52c3541d163d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8650b750-4f90-46c3-9a28-52c3541d163d" (UID: "8650b750-4f90-46c3-9a28-52c3541d163d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:45:59.187835 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:59.187811 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8650b750-4f90-46c3-9a28-52c3541d163d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8650b750-4f90-46c3-9a28-52c3541d163d" (UID: "8650b750-4f90-46c3-9a28-52c3541d163d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:45:59.189204 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:59.189166 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8650b750-4f90-46c3-9a28-52c3541d163d-kube-api-access-hl6qv" (OuterVolumeSpecName: "kube-api-access-hl6qv") pod "8650b750-4f90-46c3-9a28-52c3541d163d" (UID: "8650b750-4f90-46c3-9a28-52c3541d163d"). InnerVolumeSpecName "kube-api-access-hl6qv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 06:45:59.189317 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:59.189261 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8650b750-4f90-46c3-9a28-52c3541d163d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8650b750-4f90-46c3-9a28-52c3541d163d" (UID: "8650b750-4f90-46c3-9a28-52c3541d163d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 06:45:59.189582 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:59.189559 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8650b750-4f90-46c3-9a28-52c3541d163d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8650b750-4f90-46c3-9a28-52c3541d163d" (UID: "8650b750-4f90-46c3-9a28-52c3541d163d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:45:59.189828 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:59.189805 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8650b750-4f90-46c3-9a28-52c3541d163d-istio-token" (OuterVolumeSpecName: "istio-token") pod "8650b750-4f90-46c3-9a28-52c3541d163d" (UID: "8650b750-4f90-46c3-9a28-52c3541d163d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 06:45:59.288093 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:59.288065 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8650b750-4f90-46c3-9a28-52c3541d163d-istio-envoy\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:45:59.288093 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:59.288089 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8650b750-4f90-46c3-9a28-52c3541d163d-istio-token\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:45:59.288093 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:59.288099 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8650b750-4f90-46c3-9a28-52c3541d163d-istio-data\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:45:59.288306 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:59.288108 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hl6qv\" (UniqueName: \"kubernetes.io/projected/8650b750-4f90-46c3-9a28-52c3541d163d-kube-api-access-hl6qv\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:45:59.288306 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:59.288119 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8650b750-4f90-46c3-9a28-52c3541d163d-workload-certs\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:45:59.288306 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:59.288129 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8650b750-4f90-46c3-9a28-52c3541d163d-workload-socket\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:45:59.288306 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:59.288136 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8650b750-4f90-46c3-9a28-52c3541d163d-istio-podinfo\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:45:59.585463 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:59.585434 2567 generic.go:356] "Generic (PLEG): container finished" podID="8650b750-4f90-46c3-9a28-52c3541d163d" containerID="9cbbaf8fabdfcd119ae228c42e6656eea821039c78aeb26a0980664b5e14a4da" exitCode=0 Sep 18 06:45:59.585627 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:59.585478 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr" Sep 18 06:45:59.585627 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:59.585517 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr" event={"ID":"8650b750-4f90-46c3-9a28-52c3541d163d","Type":"ContainerDied","Data":"9cbbaf8fabdfcd119ae228c42e6656eea821039c78aeb26a0980664b5e14a4da"} Sep 18 06:45:59.585627 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:59.585583 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr" event={"ID":"8650b750-4f90-46c3-9a28-52c3541d163d","Type":"ContainerDied","Data":"0157c31a972b62f610a38916de6eaa7b95803f1ef72aa99fbebcab3b4262cf9e"} Sep 18 06:45:59.585627 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:59.585611 2567 scope.go:117] "RemoveContainer" containerID="9cbbaf8fabdfcd119ae228c42e6656eea821039c78aeb26a0980664b5e14a4da" Sep 18 06:45:59.596069 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:59.596054 2567 scope.go:117] "RemoveContainer" containerID="b82857e607e1666c0737abf89991dd370550780720115c6a5088d33a1badfb07" Sep 18 06:45:59.605235 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:59.605213 2567 scope.go:117] "RemoveContainer" containerID="9cbbaf8fabdfcd119ae228c42e6656eea821039c78aeb26a0980664b5e14a4da" Sep 18 06:45:59.605564 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:45:59.605540 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cbbaf8fabdfcd119ae228c42e6656eea821039c78aeb26a0980664b5e14a4da\": container with ID starting with 9cbbaf8fabdfcd119ae228c42e6656eea821039c78aeb26a0980664b5e14a4da not found: ID does not exist" containerID="9cbbaf8fabdfcd119ae228c42e6656eea821039c78aeb26a0980664b5e14a4da" Sep 18 06:45:59.605672 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:59.605575 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cbbaf8fabdfcd119ae228c42e6656eea821039c78aeb26a0980664b5e14a4da"} err="failed to get container status \"9cbbaf8fabdfcd119ae228c42e6656eea821039c78aeb26a0980664b5e14a4da\": rpc error: code = NotFound desc = could not find container \"9cbbaf8fabdfcd119ae228c42e6656eea821039c78aeb26a0980664b5e14a4da\": container with ID starting with 9cbbaf8fabdfcd119ae228c42e6656eea821039c78aeb26a0980664b5e14a4da not found: ID does not exist" Sep 18 06:45:59.605672 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:59.605598 2567 scope.go:117] "RemoveContainer" containerID="b82857e607e1666c0737abf89991dd370550780720115c6a5088d33a1badfb07" Sep 18 06:45:59.605902 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:45:59.605873 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b82857e607e1666c0737abf89991dd370550780720115c6a5088d33a1badfb07\": container with ID starting with b82857e607e1666c0737abf89991dd370550780720115c6a5088d33a1badfb07 not found: ID does not exist" containerID="b82857e607e1666c0737abf89991dd370550780720115c6a5088d33a1badfb07" Sep 18 06:45:59.605971 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:59.605911 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b82857e607e1666c0737abf89991dd370550780720115c6a5088d33a1badfb07"} err="failed to get container status \"b82857e607e1666c0737abf89991dd370550780720115c6a5088d33a1badfb07\": rpc error: code = NotFound desc = could not find container \"b82857e607e1666c0737abf89991dd370550780720115c6a5088d33a1badfb07\": container with ID starting with b82857e607e1666c0737abf89991dd370550780720115c6a5088d33a1badfb07 not found: ID does not exist" Sep 18 06:45:59.608793 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:59.608770 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr"] Sep 18 06:45:59.617489 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:45:59.614112 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d887938f-openshift-default-7d5ddb489b-bw7kr"] Sep 18 06:46:00.445915 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:00.445881 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8650b750-4f90-46c3-9a28-52c3541d163d" path="/var/lib/kubelet/pods/8650b750-4f90-46c3-9a28-52c3541d163d/volumes" Sep 18 06:46:07.127393 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:07.127297 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp"] Sep 18 06:46:07.127772 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:07.127758 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8650b750-4f90-46c3-9a28-52c3541d163d" containerName="istio-proxy" Sep 18 06:46:07.127818 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:07.127774 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8650b750-4f90-46c3-9a28-52c3541d163d" containerName="istio-proxy" Sep 18 06:46:07.127818 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:07.127790 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="046679b1-2638-42e1-ace9-582e569abdbb" containerName="payload-pre-processing" Sep 18 06:46:07.127818 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:07.127797 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="046679b1-2638-42e1-ace9-582e569abdbb" containerName="payload-pre-processing" Sep 18 06:46:07.127818 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:07.127813 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68671084-e485-4b2f-9b88-9ee70253314f" containerName="revoke-keys" Sep 18 06:46:07.127818 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:07.127819 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="68671084-e485-4b2f-9b88-9ee70253314f" containerName="revoke-keys" Sep 18 06:46:07.128055 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:07.127906 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="046679b1-2638-42e1-ace9-582e569abdbb" containerName="payload-pre-processing" Sep 18 06:46:07.128055 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:07.127917 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="68671084-e485-4b2f-9b88-9ee70253314f" containerName="revoke-keys" Sep 18 06:46:07.128055 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:07.127928 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8650b750-4f90-46c3-9a28-52c3541d163d" containerName="istio-proxy" Sep 18 06:46:07.132904 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:07.132876 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp" Sep 18 06:46:07.135516 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:07.135491 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-bac224-openshift-default-dockercfg-65xtp\"" Sep 18 06:46:07.146116 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:07.146093 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp"] Sep 18 06:46:07.153877 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:07.153851 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d74f3c3-8297-4e77-9223-5bba5a963fea-istio-data\") pod \"e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp\" (UID: \"1d74f3c3-8297-4e77-9223-5bba5a963fea\") " pod="openshift-ingress/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp" Sep 18 06:46:07.153982 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:07.153915 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d74f3c3-8297-4e77-9223-5bba5a963fea-istio-envoy\") pod \"e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp\" (UID: \"1d74f3c3-8297-4e77-9223-5bba5a963fea\") " pod="openshift-ingress/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp" Sep 18 06:46:07.154030 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:07.153972 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d74f3c3-8297-4e77-9223-5bba5a963fea-istiod-ca-cert\") pod \"e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp\" (UID: \"1d74f3c3-8297-4e77-9223-5bba5a963fea\") " pod="openshift-ingress/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp" Sep 18 06:46:07.154076 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:07.154059 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d74f3c3-8297-4e77-9223-5bba5a963fea-workload-socket\") pod \"e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp\" (UID: \"1d74f3c3-8297-4e77-9223-5bba5a963fea\") " pod="openshift-ingress/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp" Sep 18 06:46:07.154113 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:07.154095 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d74f3c3-8297-4e77-9223-5bba5a963fea-workload-certs\") pod \"e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp\" (UID: \"1d74f3c3-8297-4e77-9223-5bba5a963fea\") " pod="openshift-ingress/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp" Sep 18 06:46:07.154148 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:07.154128 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d74f3c3-8297-4e77-9223-5bba5a963fea-credential-socket\") pod \"e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp\" (UID: \"1d74f3c3-8297-4e77-9223-5bba5a963fea\") " pod="openshift-ingress/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp" Sep 18 06:46:07.154182 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:07.154163 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d74f3c3-8297-4e77-9223-5bba5a963fea-istio-token\") pod \"e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp\" (UID: \"1d74f3c3-8297-4e77-9223-5bba5a963fea\") " pod="openshift-ingress/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp" Sep 18 06:46:07.154217 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:07.154191 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2mg4\" (UniqueName: \"kubernetes.io/projected/1d74f3c3-8297-4e77-9223-5bba5a963fea-kube-api-access-n2mg4\") pod \"e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp\" (UID: \"1d74f3c3-8297-4e77-9223-5bba5a963fea\") " pod="openshift-ingress/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp" Sep 18 06:46:07.154259 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:07.154234 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d74f3c3-8297-4e77-9223-5bba5a963fea-istio-podinfo\") pod \"e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp\" (UID: \"1d74f3c3-8297-4e77-9223-5bba5a963fea\") " pod="openshift-ingress/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp" Sep 18 06:46:07.255336 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:07.255290 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d74f3c3-8297-4e77-9223-5bba5a963fea-istio-podinfo\") pod \"e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp\" (UID: \"1d74f3c3-8297-4e77-9223-5bba5a963fea\") " pod="openshift-ingress/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp" Sep 18 06:46:07.255584 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:07.255391 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d74f3c3-8297-4e77-9223-5bba5a963fea-istio-data\") pod \"e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp\" (UID: \"1d74f3c3-8297-4e77-9223-5bba5a963fea\") " pod="openshift-ingress/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp" Sep 18 06:46:07.255584 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:07.255501 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d74f3c3-8297-4e77-9223-5bba5a963fea-istio-envoy\") pod \"e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp\" (UID: \"1d74f3c3-8297-4e77-9223-5bba5a963fea\") " pod="openshift-ingress/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp" Sep 18 06:46:07.255584 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:07.255534 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d74f3c3-8297-4e77-9223-5bba5a963fea-istiod-ca-cert\") pod \"e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp\" (UID: \"1d74f3c3-8297-4e77-9223-5bba5a963fea\") " pod="openshift-ingress/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp" Sep 18 06:46:07.255764 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:07.255618 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d74f3c3-8297-4e77-9223-5bba5a963fea-workload-socket\") pod \"e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp\" (UID: \"1d74f3c3-8297-4e77-9223-5bba5a963fea\") " pod="openshift-ingress/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp" Sep 18 06:46:07.255764 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:07.255649 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d74f3c3-8297-4e77-9223-5bba5a963fea-workload-certs\") pod \"e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp\" (UID: \"1d74f3c3-8297-4e77-9223-5bba5a963fea\") " pod="openshift-ingress/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp" Sep 18 06:46:07.255764 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:07.255679 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d74f3c3-8297-4e77-9223-5bba5a963fea-credential-socket\") pod \"e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp\" (UID: \"1d74f3c3-8297-4e77-9223-5bba5a963fea\") " pod="openshift-ingress/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp" Sep 18 06:46:07.255764 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:07.255718 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d74f3c3-8297-4e77-9223-5bba5a963fea-istio-token\") pod \"e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp\" (UID: \"1d74f3c3-8297-4e77-9223-5bba5a963fea\") " pod="openshift-ingress/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp" Sep 18 06:46:07.255764 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:07.255749 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2mg4\" (UniqueName: \"kubernetes.io/projected/1d74f3c3-8297-4e77-9223-5bba5a963fea-kube-api-access-n2mg4\") pod \"e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp\" (UID: \"1d74f3c3-8297-4e77-9223-5bba5a963fea\") " pod="openshift-ingress/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp" Sep 18 06:46:07.257010 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:07.256966 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d74f3c3-8297-4e77-9223-5bba5a963fea-workload-socket\") pod \"e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp\" (UID: \"1d74f3c3-8297-4e77-9223-5bba5a963fea\") " pod="openshift-ingress/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp" Sep 18 06:46:07.257259 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:07.257215 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d74f3c3-8297-4e77-9223-5bba5a963fea-credential-socket\") pod \"e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp\" (UID: \"1d74f3c3-8297-4e77-9223-5bba5a963fea\") " pod="openshift-ingress/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp" Sep 18 06:46:07.257358 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:07.257329 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d74f3c3-8297-4e77-9223-5bba5a963fea-istiod-ca-cert\") pod \"e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp\" (UID: \"1d74f3c3-8297-4e77-9223-5bba5a963fea\") " pod="openshift-ingress/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp" Sep 18 06:46:07.257358 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:07.257336 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d74f3c3-8297-4e77-9223-5bba5a963fea-istio-data\") pod \"e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp\" (UID: \"1d74f3c3-8297-4e77-9223-5bba5a963fea\") " pod="openshift-ingress/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp" Sep 18 06:46:07.257566 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:07.257514 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d74f3c3-8297-4e77-9223-5bba5a963fea-workload-certs\") pod \"e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp\" (UID: \"1d74f3c3-8297-4e77-9223-5bba5a963fea\") " pod="openshift-ingress/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp" Sep 18 06:46:07.260210 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:07.260181 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d74f3c3-8297-4e77-9223-5bba5a963fea-istio-envoy\") pod \"e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp\" (UID: \"1d74f3c3-8297-4e77-9223-5bba5a963fea\") " pod="openshift-ingress/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp" Sep 18 06:46:07.260433 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:07.260190 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d74f3c3-8297-4e77-9223-5bba5a963fea-istio-podinfo\") pod \"e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp\" (UID: \"1d74f3c3-8297-4e77-9223-5bba5a963fea\") " pod="openshift-ingress/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp" Sep 18 06:46:07.268112 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:07.268090 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2mg4\" (UniqueName: \"kubernetes.io/projected/1d74f3c3-8297-4e77-9223-5bba5a963fea-kube-api-access-n2mg4\") pod \"e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp\" (UID: \"1d74f3c3-8297-4e77-9223-5bba5a963fea\") " pod="openshift-ingress/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp" Sep 18 06:46:07.270625 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:07.270601 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d74f3c3-8297-4e77-9223-5bba5a963fea-istio-token\") pod \"e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp\" (UID: \"1d74f3c3-8297-4e77-9223-5bba5a963fea\") " pod="openshift-ingress/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp" Sep 18 06:46:07.443812 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:07.443745 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp" Sep 18 06:46:07.730970 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:07.730940 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp"] Sep 18 06:46:07.732340 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:46:07.732306 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d74f3c3_8297_4e77_9223_5bba5a963fea.slice/crio-f06bc7f73820427928b92b3ac7c6f8a155f3a9889bcadfc977ed05556837c2b6 WatchSource:0}: Error finding container f06bc7f73820427928b92b3ac7c6f8a155f3a9889bcadfc977ed05556837c2b6: Status 404 returned error can't find the container with id f06bc7f73820427928b92b3ac7c6f8a155f3a9889bcadfc977ed05556837c2b6 Sep 18 06:46:07.737476 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:07.737437 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:46:07.737587 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:07.737514 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:46:07.737587 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:07.737551 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:46:08.642237 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:08.642207 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp" event={"ID":"1d74f3c3-8297-4e77-9223-5bba5a963fea","Type":"ContainerStarted","Data":"8f587b2dd77700a58ee42e3b2eb5fa4aac0e1280b2795c8519cffb78ff5e89cc"} Sep 18 06:46:08.642630 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:08.642256 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp" event={"ID":"1d74f3c3-8297-4e77-9223-5bba5a963fea","Type":"ContainerStarted","Data":"f06bc7f73820427928b92b3ac7c6f8a155f3a9889bcadfc977ed05556837c2b6"} Sep 18 06:46:08.662697 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:08.662654 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp" podStartSLOduration=1.662638132 podStartE2EDuration="1.662638132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:46:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:46:08.66057818 +0000 UTC m=+1430.399994394" watchObservedRunningTime="2026-09-18 06:46:08.662638132 +0000 UTC m=+1430.402054306" Sep 18 06:46:09.444894 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:09.444864 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp" Sep 18 06:46:09.450219 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:09.450195 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp" Sep 18 06:46:09.648969 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:09.648937 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp" Sep 18 06:46:09.649630 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:09.649611 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp" Sep 18 06:46:14.740645 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:14.740610 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bac224-6b5977dbc4-wrfrp"] Sep 18 06:46:14.741062 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:14.741049 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8650b750-4f90-46c3-9a28-52c3541d163d" containerName="istio-proxy" Sep 18 06:46:14.741062 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:14.741063 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8650b750-4f90-46c3-9a28-52c3541d163d" containerName="istio-proxy" Sep 18 06:46:14.741148 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:14.741139 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8650b750-4f90-46c3-9a28-52c3541d163d" containerName="istio-proxy" Sep 18 06:46:14.744498 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:14.744481 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bac224-6b5977dbc4-wrfrp" Sep 18 06:46:14.746944 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:14.746921 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-bac224-dockercfg-tcw8r\"" Sep 18 06:46:14.747077 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:14.746924 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-bac224\"" Sep 18 06:46:14.757636 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:14.757602 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bac224-6b5977dbc4-wrfrp"] Sep 18 06:46:14.770440 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:14.770411 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-bac224-58676c465c-2j9m2"] Sep 18 06:46:14.774008 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:14.773991 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bac224-58676c465c-2j9m2" Sep 18 06:46:14.787422 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:14.787392 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-bac224-58676c465c-2j9m2"] Sep 18 06:46:14.835096 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:14.833118 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53043d19-d768-4926-9193-ba48cf5508da-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-bac224-58676c465c-2j9m2\" (UID: \"53043d19-d768-4926-9193-ba48cf5508da\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bac224-58676c465c-2j9m2" Sep 18 06:46:14.835096 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:14.833185 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7gdt\" (UniqueName: \"kubernetes.io/projected/53043d19-d768-4926-9193-ba48cf5508da-kube-api-access-p7gdt\") pod \"payload-processing-e2e-shared-a-w2-bac224-58676c465c-2j9m2\" (UID: \"53043d19-d768-4926-9193-ba48cf5508da\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bac224-58676c465c-2j9m2" Sep 18 06:46:14.835096 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:14.833313 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3130492d-c8d9-453b-bd48-13422815eb42-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-bac224-6b5977dbc4-wrfrp\" (UID: \"3130492d-c8d9-453b-bd48-13422815eb42\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bac224-6b5977dbc4-wrfrp" Sep 18 06:46:14.835096 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:14.833356 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fshl5\" (UniqueName: \"kubernetes.io/projected/3130492d-c8d9-453b-bd48-13422815eb42-kube-api-access-fshl5\") pod \"payload-pre-processing-e2e-shared-a-w2-bac224-6b5977dbc4-wrfrp\" (UID: \"3130492d-c8d9-453b-bd48-13422815eb42\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bac224-6b5977dbc4-wrfrp" Sep 18 06:46:14.934723 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:14.934685 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3130492d-c8d9-453b-bd48-13422815eb42-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-bac224-6b5977dbc4-wrfrp\" (UID: \"3130492d-c8d9-453b-bd48-13422815eb42\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bac224-6b5977dbc4-wrfrp" Sep 18 06:46:14.934894 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:14.934730 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fshl5\" (UniqueName: \"kubernetes.io/projected/3130492d-c8d9-453b-bd48-13422815eb42-kube-api-access-fshl5\") pod \"payload-pre-processing-e2e-shared-a-w2-bac224-6b5977dbc4-wrfrp\" (UID: \"3130492d-c8d9-453b-bd48-13422815eb42\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bac224-6b5977dbc4-wrfrp" Sep 18 06:46:14.934894 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:14.934807 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53043d19-d768-4926-9193-ba48cf5508da-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-bac224-58676c465c-2j9m2\" (UID: \"53043d19-d768-4926-9193-ba48cf5508da\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bac224-58676c465c-2j9m2" Sep 18 06:46:14.934894 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:14.934836 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7gdt\" (UniqueName: \"kubernetes.io/projected/53043d19-d768-4926-9193-ba48cf5508da-kube-api-access-p7gdt\") pod \"payload-processing-e2e-shared-a-w2-bac224-58676c465c-2j9m2\" (UID: \"53043d19-d768-4926-9193-ba48cf5508da\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bac224-58676c465c-2j9m2" Sep 18 06:46:14.935431 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:14.935407 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3130492d-c8d9-453b-bd48-13422815eb42-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-bac224-6b5977dbc4-wrfrp\" (UID: \"3130492d-c8d9-453b-bd48-13422815eb42\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bac224-6b5977dbc4-wrfrp" Sep 18 06:46:14.935592 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:14.935575 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53043d19-d768-4926-9193-ba48cf5508da-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-bac224-58676c465c-2j9m2\" (UID: \"53043d19-d768-4926-9193-ba48cf5508da\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bac224-58676c465c-2j9m2" Sep 18 06:46:14.942902 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:14.942875 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fshl5\" (UniqueName: \"kubernetes.io/projected/3130492d-c8d9-453b-bd48-13422815eb42-kube-api-access-fshl5\") pod \"payload-pre-processing-e2e-shared-a-w2-bac224-6b5977dbc4-wrfrp\" (UID: \"3130492d-c8d9-453b-bd48-13422815eb42\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bac224-6b5977dbc4-wrfrp" Sep 18 06:46:14.943016 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:14.942968 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7gdt\" (UniqueName: \"kubernetes.io/projected/53043d19-d768-4926-9193-ba48cf5508da-kube-api-access-p7gdt\") pod \"payload-processing-e2e-shared-a-w2-bac224-58676c465c-2j9m2\" (UID: \"53043d19-d768-4926-9193-ba48cf5508da\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bac224-58676c465c-2j9m2" Sep 18 06:46:15.059874 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:15.059800 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bac224-6b5977dbc4-wrfrp" Sep 18 06:46:15.084982 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:15.084959 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bac224-58676c465c-2j9m2" Sep 18 06:46:15.335654 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:15.335617 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bac224-6b5977dbc4-wrfrp"] Sep 18 06:46:15.336122 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:46:15.336096 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3130492d_c8d9_453b_bd48_13422815eb42.slice/crio-6e3534a4d3ac802d3e375eaf6b636c65cbb7d704758f1f2e5a5301700aa9a720 WatchSource:0}: Error finding container 6e3534a4d3ac802d3e375eaf6b636c65cbb7d704758f1f2e5a5301700aa9a720: Status 404 returned error can't find the container with id 6e3534a4d3ac802d3e375eaf6b636c65cbb7d704758f1f2e5a5301700aa9a720 Sep 18 06:46:15.369215 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:15.369186 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-bac224-58676c465c-2j9m2"] Sep 18 06:46:15.371684 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:46:15.371655 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53043d19_d768_4926_9193_ba48cf5508da.slice/crio-4c316aebb3b67c1ea0ab301ab32afdfe84982f05c6bfcdc0e4d87dd1ed8cdc7f WatchSource:0}: Error finding container 4c316aebb3b67c1ea0ab301ab32afdfe84982f05c6bfcdc0e4d87dd1ed8cdc7f: Status 404 returned error can't find the container with id 4c316aebb3b67c1ea0ab301ab32afdfe84982f05c6bfcdc0e4d87dd1ed8cdc7f Sep 18 06:46:15.690111 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:15.690004 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bac224-58676c465c-2j9m2" event={"ID":"53043d19-d768-4926-9193-ba48cf5508da","Type":"ContainerStarted","Data":"dd8e92711717fb7cf548dfbae80d21d9faaa21fa69e36c8d93743b15e1b2f1f0"} Sep 18 06:46:15.690111 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:15.690069 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bac224-58676c465c-2j9m2" event={"ID":"53043d19-d768-4926-9193-ba48cf5508da","Type":"ContainerStarted","Data":"4c316aebb3b67c1ea0ab301ab32afdfe84982f05c6bfcdc0e4d87dd1ed8cdc7f"} Sep 18 06:46:15.690111 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:15.690100 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bac224-58676c465c-2j9m2" Sep 18 06:46:15.691842 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:15.691818 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bac224-6b5977dbc4-wrfrp" event={"ID":"3130492d-c8d9-453b-bd48-13422815eb42","Type":"ContainerStarted","Data":"feeedee4db5197541329f44c1a29656853240aa9331fd0f0e8d7048569ccc4dc"} Sep 18 06:46:15.691967 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:15.691866 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bac224-6b5977dbc4-wrfrp" event={"ID":"3130492d-c8d9-453b-bd48-13422815eb42","Type":"ContainerStarted","Data":"6e3534a4d3ac802d3e375eaf6b636c65cbb7d704758f1f2e5a5301700aa9a720"} Sep 18 06:46:15.691967 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:15.691894 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bac224-6b5977dbc4-wrfrp" Sep 18 06:46:15.707704 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:15.707664 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bac224-58676c465c-2j9m2" podStartSLOduration=1.7076495459999999 podStartE2EDuration="1.707649546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:46:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:46:15.704323355 +0000 UTC m=+1437.443739531" watchObservedRunningTime="2026-09-18 06:46:15.707649546 +0000 UTC m=+1437.447065755" Sep 18 06:46:15.719595 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:15.719551 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bac224-6b5977dbc4-wrfrp" podStartSLOduration=1.7195414549999999 podStartE2EDuration="1.719541455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:46:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:46:15.717974445 +0000 UTC m=+1437.457390644" watchObservedRunningTime="2026-09-18 06:46:15.719541455 +0000 UTC m=+1437.458957628" Sep 18 06:46:26.699645 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:26.699616 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bac224-58676c465c-2j9m2" Sep 18 06:46:26.700117 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:26.699680 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bac224-6b5977dbc4-wrfrp" Sep 18 06:46:33.290598 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:33.290546 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-14ee59-hll97"] Sep 18 06:46:33.297104 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:33.297025 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-14ee59-hll97" Sep 18 06:46:33.299444 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:33.299412 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-14ee59-hll97"] Sep 18 06:46:33.299772 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:33.299751 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 18 06:46:33.299772 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:33.299765 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-88t7h\"" Sep 18 06:46:33.412126 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:33.412097 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/939812ec-1c83-496c-9ec3-83d0f3795c58-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-14ee59-hll97\" (UID: \"939812ec-1c83-496c-9ec3-83d0f3795c58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-14ee59-hll97" Sep 18 06:46:33.513393 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:33.513327 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/939812ec-1c83-496c-9ec3-83d0f3795c58-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-14ee59-hll97\" (UID: \"939812ec-1c83-496c-9ec3-83d0f3795c58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-14ee59-hll97" Sep 18 06:46:33.514307 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:33.514283 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/939812ec-1c83-496c-9ec3-83d0f3795c58-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-14ee59-hll97\" (UID: \"939812ec-1c83-496c-9ec3-83d0f3795c58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-14ee59-hll97" Sep 18 06:46:33.609511 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:33.609480 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-14ee59-hll97" Sep 18 06:46:33.903599 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:33.903567 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-14ee59-hll97"] Sep 18 06:46:33.904468 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:46:33.904443 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod939812ec_1c83_496c_9ec3_83d0f3795c58.slice/crio-5a5b4fc227b89302a3c60ab3596c12b1342eeb14c55b50a7e9ee2ea25a156104 WatchSource:0}: Error finding container 5a5b4fc227b89302a3c60ab3596c12b1342eeb14c55b50a7e9ee2ea25a156104: Status 404 returned error can't find the container with id 5a5b4fc227b89302a3c60ab3596c12b1342eeb14c55b50a7e9ee2ea25a156104 Sep 18 06:46:34.819997 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:34.819966 2567 generic.go:356] "Generic (PLEG): container finished" podID="939812ec-1c83-496c-9ec3-83d0f3795c58" containerID="d7d3c4fb9076e5fabc3424ce3833862cc2de77cbaa25fbbc2bf27353e612e166" exitCode=0 Sep 18 06:46:34.820330 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:34.820042 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-14ee59-hll97" event={"ID":"939812ec-1c83-496c-9ec3-83d0f3795c58","Type":"ContainerDied","Data":"d7d3c4fb9076e5fabc3424ce3833862cc2de77cbaa25fbbc2bf27353e612e166"} Sep 18 06:46:34.820330 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:34.820078 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-14ee59-hll97" event={"ID":"939812ec-1c83-496c-9ec3-83d0f3795c58","Type":"ContainerStarted","Data":"5a5b4fc227b89302a3c60ab3596c12b1342eeb14c55b50a7e9ee2ea25a156104"} Sep 18 06:46:36.015259 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:36.015222 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-14ee59-hll97" Sep 18 06:46:36.138934 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:36.138900 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/939812ec-1c83-496c-9ec3-83d0f3795c58-maas-api-service-ca\") pod \"939812ec-1c83-496c-9ec3-83d0f3795c58\" (UID: \"939812ec-1c83-496c-9ec3-83d0f3795c58\") " Sep 18 06:46:36.139237 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:36.139213 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/939812ec-1c83-496c-9ec3-83d0f3795c58-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "939812ec-1c83-496c-9ec3-83d0f3795c58" (UID: "939812ec-1c83-496c-9ec3-83d0f3795c58"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 06:46:36.241566 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:36.241536 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/939812ec-1c83-496c-9ec3-83d0f3795c58-maas-api-service-ca\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:46:36.834083 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:36.834053 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-14ee59-hll97" Sep 18 06:46:36.834246 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:36.834082 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-14ee59-hll97" event={"ID":"939812ec-1c83-496c-9ec3-83d0f3795c58","Type":"ContainerDied","Data":"5a5b4fc227b89302a3c60ab3596c12b1342eeb14c55b50a7e9ee2ea25a156104"} Sep 18 06:46:36.834246 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:36.834129 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a5b4fc227b89302a3c60ab3596c12b1342eeb14c55b50a7e9ee2ea25a156104" Sep 18 06:46:39.808862 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:39.808330 2567 patch_prober.go:28] interesting pod/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": read tcp 10.134.0.2:41814->10.134.0.74:15021: read: connection reset by peer" start-of-body= Sep 18 06:46:39.808862 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:39.808447 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp" podUID="1d74f3c3-8297-4e77-9223-5bba5a963fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": read tcp 10.134.0.2:41814->10.134.0.74:15021: read: connection reset by peer" Sep 18 06:46:39.857998 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:39.857972 2567 generic.go:356] "Generic (PLEG): container finished" podID="1d74f3c3-8297-4e77-9223-5bba5a963fea" containerID="8f587b2dd77700a58ee42e3b2eb5fa4aac0e1280b2795c8519cffb78ff5e89cc" exitCode=0 Sep 18 06:46:39.858113 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:39.858035 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp" event={"ID":"1d74f3c3-8297-4e77-9223-5bba5a963fea","Type":"ContainerDied","Data":"8f587b2dd77700a58ee42e3b2eb5fa4aac0e1280b2795c8519cffb78ff5e89cc"} Sep 18 06:46:39.858434 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:39.858417 2567 scope.go:117] "RemoveContainer" containerID="8f587b2dd77700a58ee42e3b2eb5fa4aac0e1280b2795c8519cffb78ff5e89cc" Sep 18 06:46:39.859652 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:39.859622 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:46:39.859753 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:39.859713 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:46:39.859753 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:39.859745 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:46:40.871788 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:40.871741 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp" event={"ID":"1d74f3c3-8297-4e77-9223-5bba5a963fea","Type":"ContainerStarted","Data":"593f1f924f7fc815fc1122f6c46f87bc05280aed4f7c591ec72fcdbec86a7b26"} Sep 18 06:46:41.444745 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:41.444706 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp" Sep 18 06:46:41.444949 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:41.444926 2567 patch_prober.go:28] interesting pod/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Sep 18 06:46:41.444999 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:41.444975 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp" podUID="1d74f3c3-8297-4e77-9223-5bba5a963fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Sep 18 06:46:42.446324 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:42.446291 2567 patch_prober.go:28] interesting pod/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Sep 18 06:46:42.446849 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:42.446352 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp" podUID="1d74f3c3-8297-4e77-9223-5bba5a963fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Sep 18 06:46:43.444156 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:43.444122 2567 patch_prober.go:28] interesting pod/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Sep 18 06:46:43.444329 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:43.444185 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp" podUID="1d74f3c3-8297-4e77-9223-5bba5a963fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Sep 18 06:46:44.444775 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:44.444738 2567 patch_prober.go:28] interesting pod/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Sep 18 06:46:44.445174 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:44.444798 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp" podUID="1d74f3c3-8297-4e77-9223-5bba5a963fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Sep 18 06:46:45.444293 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:45.444261 2567 patch_prober.go:28] interesting pod/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Sep 18 06:46:45.444476 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:45.444309 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp" podUID="1d74f3c3-8297-4e77-9223-5bba5a963fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Sep 18 06:46:46.448627 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:46.448596 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp" Sep 18 06:46:46.449015 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:46.448838 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp" Sep 18 06:46:46.449624 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:46.449607 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp" Sep 18 06:46:47.581201 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:47.581163 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-14ee59-29828565-l7l28"] Sep 18 06:46:47.583946 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:47.583924 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-14ee59-29828565-l7l28"] Sep 18 06:46:47.610306 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:47.610271 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-14ee59-7784b6b66f-2bcg9"] Sep 18 06:46:47.610550 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:47.610525 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-14ee59-7784b6b66f-2bcg9" podUID="a81a0ebf-f60e-4995-ba97-cfa65c37e81b" containerName="payload-processing" containerID="cri-o://606c3f1f4d9f58b94bc0029110da6a1190d1f93151b30c74fb2b5e49e8ae2a17" gracePeriod=30 Sep 18 06:46:47.618270 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:47.617806 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-14ee59-7b9c6754db-58xcm"] Sep 18 06:46:47.618810 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:47.618313 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-14ee59-7b9c6754db-58xcm" podUID="66c9823b-6600-467a-b1c2-20800910a671" containerName="payload-pre-processing" containerID="cri-o://ef3b72d204d76c813316ecb79077b3a51678a2a915d0438fb72439c2ece4a1e8" gracePeriod=30 Sep 18 06:46:47.766533 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:47.766493 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-14ee59-hll97"] Sep 18 06:46:47.771505 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:47.771479 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-14ee59-hll97"] Sep 18 06:46:47.887174 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:47.887147 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-14ee59-7784b6b66f-2bcg9" Sep 18 06:46:47.907627 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:47.907605 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-14ee59-7b9c6754db-58xcm" Sep 18 06:46:47.921631 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:47.921603 2567 generic.go:356] "Generic (PLEG): container finished" podID="66c9823b-6600-467a-b1c2-20800910a671" containerID="ef3b72d204d76c813316ecb79077b3a51678a2a915d0438fb72439c2ece4a1e8" exitCode=0 Sep 18 06:46:47.921732 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:47.921645 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-14ee59-7b9c6754db-58xcm" Sep 18 06:46:47.921732 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:47.921651 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-14ee59-7b9c6754db-58xcm" event={"ID":"66c9823b-6600-467a-b1c2-20800910a671","Type":"ContainerDied","Data":"ef3b72d204d76c813316ecb79077b3a51678a2a915d0438fb72439c2ece4a1e8"} Sep 18 06:46:47.921732 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:47.921712 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-14ee59-7b9c6754db-58xcm" event={"ID":"66c9823b-6600-467a-b1c2-20800910a671","Type":"ContainerDied","Data":"1fa8ee41c7fa9df098fed817efc867e2e461dd301c0bd3afd5813099c2db0e37"} Sep 18 06:46:47.921910 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:47.921752 2567 scope.go:117] "RemoveContainer" containerID="ef3b72d204d76c813316ecb79077b3a51678a2a915d0438fb72439c2ece4a1e8" Sep 18 06:46:47.924399 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:47.924346 2567 generic.go:356] "Generic (PLEG): container finished" podID="a81a0ebf-f60e-4995-ba97-cfa65c37e81b" containerID="606c3f1f4d9f58b94bc0029110da6a1190d1f93151b30c74fb2b5e49e8ae2a17" exitCode=0 Sep 18 06:46:47.924502 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:47.924428 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-14ee59-7784b6b66f-2bcg9" Sep 18 06:46:47.924502 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:47.924445 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-14ee59-7784b6b66f-2bcg9" event={"ID":"a81a0ebf-f60e-4995-ba97-cfa65c37e81b","Type":"ContainerDied","Data":"606c3f1f4d9f58b94bc0029110da6a1190d1f93151b30c74fb2b5e49e8ae2a17"} Sep 18 06:46:47.924502 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:47.924499 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-14ee59-7784b6b66f-2bcg9" event={"ID":"a81a0ebf-f60e-4995-ba97-cfa65c37e81b","Type":"ContainerDied","Data":"4f093d3a2eeb775b32076d2aca66662370fa23c74d11272a491fa056bb5ed1f2"} Sep 18 06:46:47.933509 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:47.933490 2567 scope.go:117] "RemoveContainer" containerID="ef3b72d204d76c813316ecb79077b3a51678a2a915d0438fb72439c2ece4a1e8" Sep 18 06:46:47.933791 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:46:47.933772 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef3b72d204d76c813316ecb79077b3a51678a2a915d0438fb72439c2ece4a1e8\": container with ID starting with ef3b72d204d76c813316ecb79077b3a51678a2a915d0438fb72439c2ece4a1e8 not found: ID does not exist" containerID="ef3b72d204d76c813316ecb79077b3a51678a2a915d0438fb72439c2ece4a1e8" Sep 18 06:46:47.933865 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:47.933799 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef3b72d204d76c813316ecb79077b3a51678a2a915d0438fb72439c2ece4a1e8"} err="failed to get container status \"ef3b72d204d76c813316ecb79077b3a51678a2a915d0438fb72439c2ece4a1e8\": rpc error: code = NotFound desc = could not find container \"ef3b72d204d76c813316ecb79077b3a51678a2a915d0438fb72439c2ece4a1e8\": container with ID starting with ef3b72d204d76c813316ecb79077b3a51678a2a915d0438fb72439c2ece4a1e8 not found: ID does not exist" Sep 18 06:46:47.933865 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:47.933816 2567 scope.go:117] "RemoveContainer" containerID="606c3f1f4d9f58b94bc0029110da6a1190d1f93151b30c74fb2b5e49e8ae2a17" Sep 18 06:46:47.942802 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:47.942785 2567 scope.go:117] "RemoveContainer" containerID="606c3f1f4d9f58b94bc0029110da6a1190d1f93151b30c74fb2b5e49e8ae2a17" Sep 18 06:46:47.943076 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:46:47.943058 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"606c3f1f4d9f58b94bc0029110da6a1190d1f93151b30c74fb2b5e49e8ae2a17\": container with ID starting with 606c3f1f4d9f58b94bc0029110da6a1190d1f93151b30c74fb2b5e49e8ae2a17 not found: ID does not exist" containerID="606c3f1f4d9f58b94bc0029110da6a1190d1f93151b30c74fb2b5e49e8ae2a17" Sep 18 06:46:47.943146 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:47.943086 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"606c3f1f4d9f58b94bc0029110da6a1190d1f93151b30c74fb2b5e49e8ae2a17"} err="failed to get container status \"606c3f1f4d9f58b94bc0029110da6a1190d1f93151b30c74fb2b5e49e8ae2a17\": rpc error: code = NotFound desc = could not find container \"606c3f1f4d9f58b94bc0029110da6a1190d1f93151b30c74fb2b5e49e8ae2a17\": container with ID starting with 606c3f1f4d9f58b94bc0029110da6a1190d1f93151b30c74fb2b5e49e8ae2a17 not found: ID does not exist" Sep 18 06:46:47.968591 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:47.968557 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a81a0ebf-f60e-4995-ba97-cfa65c37e81b-plugins-config-volume\") pod \"a81a0ebf-f60e-4995-ba97-cfa65c37e81b\" (UID: \"a81a0ebf-f60e-4995-ba97-cfa65c37e81b\") " Sep 18 06:46:47.968779 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:47.968606 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bncdn\" (UniqueName: \"kubernetes.io/projected/66c9823b-6600-467a-b1c2-20800910a671-kube-api-access-bncdn\") pod \"66c9823b-6600-467a-b1c2-20800910a671\" (UID: \"66c9823b-6600-467a-b1c2-20800910a671\") " Sep 18 06:46:47.968779 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:47.968709 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66c9823b-6600-467a-b1c2-20800910a671-plugins-config-volume\") pod \"66c9823b-6600-467a-b1c2-20800910a671\" (UID: \"66c9823b-6600-467a-b1c2-20800910a671\") " Sep 18 06:46:47.968779 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:47.968761 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5lmrk\" (UniqueName: \"kubernetes.io/projected/a81a0ebf-f60e-4995-ba97-cfa65c37e81b-kube-api-access-5lmrk\") pod \"a81a0ebf-f60e-4995-ba97-cfa65c37e81b\" (UID: \"a81a0ebf-f60e-4995-ba97-cfa65c37e81b\") " Sep 18 06:46:47.969262 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:47.969103 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66c9823b-6600-467a-b1c2-20800910a671-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "66c9823b-6600-467a-b1c2-20800910a671" (UID: "66c9823b-6600-467a-b1c2-20800910a671"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 06:46:47.971280 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:47.969343 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a81a0ebf-f60e-4995-ba97-cfa65c37e81b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a81a0ebf-f60e-4995-ba97-cfa65c37e81b" (UID: "a81a0ebf-f60e-4995-ba97-cfa65c37e81b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 06:46:47.971280 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:47.971193 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66c9823b-6600-467a-b1c2-20800910a671-plugins-config-volume\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:46:47.971280 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:47.971214 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a81a0ebf-f60e-4995-ba97-cfa65c37e81b-plugins-config-volume\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:46:47.972807 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:47.972474 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66c9823b-6600-467a-b1c2-20800910a671-kube-api-access-bncdn" (OuterVolumeSpecName: "kube-api-access-bncdn") pod "66c9823b-6600-467a-b1c2-20800910a671" (UID: "66c9823b-6600-467a-b1c2-20800910a671"). InnerVolumeSpecName "kube-api-access-bncdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 06:46:47.972903 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:47.972823 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a81a0ebf-f60e-4995-ba97-cfa65c37e81b-kube-api-access-5lmrk" (OuterVolumeSpecName: "kube-api-access-5lmrk") pod "a81a0ebf-f60e-4995-ba97-cfa65c37e81b" (UID: "a81a0ebf-f60e-4995-ba97-cfa65c37e81b"). InnerVolumeSpecName "kube-api-access-5lmrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 06:46:48.072693 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:48.072662 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lmrk\" (UniqueName: \"kubernetes.io/projected/a81a0ebf-f60e-4995-ba97-cfa65c37e81b-kube-api-access-5lmrk\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:46:48.072693 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:48.072687 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bncdn\" (UniqueName: \"kubernetes.io/projected/66c9823b-6600-467a-b1c2-20800910a671-kube-api-access-bncdn\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:46:48.245111 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:48.245069 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-14ee59-7b9c6754db-58xcm"] Sep 18 06:46:48.248532 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:48.248509 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-14ee59-7b9c6754db-58xcm"] Sep 18 06:46:48.258319 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:48.258297 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-14ee59-7784b6b66f-2bcg9"] Sep 18 06:46:48.261873 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:48.261850 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-14ee59-7784b6b66f-2bcg9"] Sep 18 06:46:48.423518 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:48.423484 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-87d618-6c8d8db6bb-m75bv"] Sep 18 06:46:48.424119 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:48.424096 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a81a0ebf-f60e-4995-ba97-cfa65c37e81b" containerName="payload-processing" Sep 18 06:46:48.424182 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:48.424124 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a81a0ebf-f60e-4995-ba97-cfa65c37e81b" containerName="payload-processing" Sep 18 06:46:48.424182 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:48.424134 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="939812ec-1c83-496c-9ec3-83d0f3795c58" containerName="revoke-keys" Sep 18 06:46:48.424182 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:48.424140 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="939812ec-1c83-496c-9ec3-83d0f3795c58" containerName="revoke-keys" Sep 18 06:46:48.424182 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:48.424148 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66c9823b-6600-467a-b1c2-20800910a671" containerName="payload-pre-processing" Sep 18 06:46:48.424182 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:48.424153 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="66c9823b-6600-467a-b1c2-20800910a671" containerName="payload-pre-processing" Sep 18 06:46:48.424347 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:48.424236 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="66c9823b-6600-467a-b1c2-20800910a671" containerName="payload-pre-processing" Sep 18 06:46:48.424347 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:48.424245 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="939812ec-1c83-496c-9ec3-83d0f3795c58" containerName="revoke-keys" Sep 18 06:46:48.424347 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:48.424255 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a81a0ebf-f60e-4995-ba97-cfa65c37e81b" containerName="payload-processing" Sep 18 06:46:48.427803 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:48.427785 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-87d618-6c8d8db6bb-m75bv" Sep 18 06:46:48.430552 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:48.430518 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-87d618\"" Sep 18 06:46:48.430666 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:48.430565 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-87d618-dockercfg-586vx\"" Sep 18 06:46:48.441975 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:48.441412 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-87d618-6c8d8db6bb-m75bv"] Sep 18 06:46:48.457807 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:48.457777 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66c9823b-6600-467a-b1c2-20800910a671" path="/var/lib/kubelet/pods/66c9823b-6600-467a-b1c2-20800910a671/volumes" Sep 18 06:46:48.458132 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:48.458117 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="939812ec-1c83-496c-9ec3-83d0f3795c58" path="/var/lib/kubelet/pods/939812ec-1c83-496c-9ec3-83d0f3795c58/volumes" Sep 18 06:46:48.458487 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:48.458474 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a81a0ebf-f60e-4995-ba97-cfa65c37e81b" path="/var/lib/kubelet/pods/a81a0ebf-f60e-4995-ba97-cfa65c37e81b/volumes" Sep 18 06:46:48.458740 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:48.458729 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be813339-652d-4bff-8403-415a0346ae0e" path="/var/lib/kubelet/pods/be813339-652d-4bff-8403-415a0346ae0e/volumes" Sep 18 06:46:48.467427 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:48.467398 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-87d618-5d94f66f4c-kwt2c"] Sep 18 06:46:48.471486 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:48.471472 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-87d618-5d94f66f4c-kwt2c" Sep 18 06:46:48.475959 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:48.475932 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2z2w\" (UniqueName: \"kubernetes.io/projected/b318b9a5-4a0a-4b3b-aa75-bc7879cebd45-kube-api-access-j2z2w\") pod \"payload-pre-processing-e2e-shared-b-w2-87d618-6c8d8db6bb-m75bv\" (UID: \"b318b9a5-4a0a-4b3b-aa75-bc7879cebd45\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-87d618-6c8d8db6bb-m75bv" Sep 18 06:46:48.476125 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:48.476076 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b318b9a5-4a0a-4b3b-aa75-bc7879cebd45-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-87d618-6c8d8db6bb-m75bv\" (UID: \"b318b9a5-4a0a-4b3b-aa75-bc7879cebd45\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-87d618-6c8d8db6bb-m75bv" Sep 18 06:46:48.479823 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:48.479803 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-87d618-5d94f66f4c-kwt2c"] Sep 18 06:46:48.577623 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:48.577538 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b318b9a5-4a0a-4b3b-aa75-bc7879cebd45-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-87d618-6c8d8db6bb-m75bv\" (UID: \"b318b9a5-4a0a-4b3b-aa75-bc7879cebd45\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-87d618-6c8d8db6bb-m75bv" Sep 18 06:46:48.577771 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:48.577654 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2z2w\" (UniqueName: \"kubernetes.io/projected/b318b9a5-4a0a-4b3b-aa75-bc7879cebd45-kube-api-access-j2z2w\") pod \"payload-pre-processing-e2e-shared-b-w2-87d618-6c8d8db6bb-m75bv\" (UID: \"b318b9a5-4a0a-4b3b-aa75-bc7879cebd45\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-87d618-6c8d8db6bb-m75bv" Sep 18 06:46:48.577771 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:48.577709 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddzbn\" (UniqueName: \"kubernetes.io/projected/584f37b8-a704-4ba3-bba0-1035c9365bda-kube-api-access-ddzbn\") pod \"payload-processing-e2e-shared-b-w2-87d618-5d94f66f4c-kwt2c\" (UID: \"584f37b8-a704-4ba3-bba0-1035c9365bda\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-87d618-5d94f66f4c-kwt2c" Sep 18 06:46:48.577846 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:48.577776 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/584f37b8-a704-4ba3-bba0-1035c9365bda-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-87d618-5d94f66f4c-kwt2c\" (UID: \"584f37b8-a704-4ba3-bba0-1035c9365bda\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-87d618-5d94f66f4c-kwt2c" Sep 18 06:46:48.578311 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:48.578292 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b318b9a5-4a0a-4b3b-aa75-bc7879cebd45-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-87d618-6c8d8db6bb-m75bv\" (UID: \"b318b9a5-4a0a-4b3b-aa75-bc7879cebd45\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-87d618-6c8d8db6bb-m75bv" Sep 18 06:46:48.585469 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:48.585442 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2z2w\" (UniqueName: \"kubernetes.io/projected/b318b9a5-4a0a-4b3b-aa75-bc7879cebd45-kube-api-access-j2z2w\") pod \"payload-pre-processing-e2e-shared-b-w2-87d618-6c8d8db6bb-m75bv\" (UID: \"b318b9a5-4a0a-4b3b-aa75-bc7879cebd45\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-87d618-6c8d8db6bb-m75bv" Sep 18 06:46:48.678435 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:48.678409 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddzbn\" (UniqueName: \"kubernetes.io/projected/584f37b8-a704-4ba3-bba0-1035c9365bda-kube-api-access-ddzbn\") pod \"payload-processing-e2e-shared-b-w2-87d618-5d94f66f4c-kwt2c\" (UID: \"584f37b8-a704-4ba3-bba0-1035c9365bda\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-87d618-5d94f66f4c-kwt2c" Sep 18 06:46:48.678551 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:48.678460 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/584f37b8-a704-4ba3-bba0-1035c9365bda-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-87d618-5d94f66f4c-kwt2c\" (UID: \"584f37b8-a704-4ba3-bba0-1035c9365bda\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-87d618-5d94f66f4c-kwt2c" Sep 18 06:46:48.679071 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:48.679054 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/584f37b8-a704-4ba3-bba0-1035c9365bda-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-87d618-5d94f66f4c-kwt2c\" (UID: \"584f37b8-a704-4ba3-bba0-1035c9365bda\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-87d618-5d94f66f4c-kwt2c" Sep 18 06:46:48.686141 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:48.686111 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddzbn\" (UniqueName: \"kubernetes.io/projected/584f37b8-a704-4ba3-bba0-1035c9365bda-kube-api-access-ddzbn\") pod \"payload-processing-e2e-shared-b-w2-87d618-5d94f66f4c-kwt2c\" (UID: \"584f37b8-a704-4ba3-bba0-1035c9365bda\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-87d618-5d94f66f4c-kwt2c" Sep 18 06:46:48.744289 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:48.744266 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-87d618-6c8d8db6bb-m75bv" Sep 18 06:46:48.783220 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:48.783189 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-87d618-5d94f66f4c-kwt2c" Sep 18 06:46:49.036219 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:49.036197 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-87d618-6c8d8db6bb-m75bv"] Sep 18 06:46:49.037909 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:46:49.037881 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb318b9a5_4a0a_4b3b_aa75_bc7879cebd45.slice/crio-b40d0281b031cb243624f66011fdcac6bf195a60edc79d7159c60585c2dcdf96 WatchSource:0}: Error finding container b40d0281b031cb243624f66011fdcac6bf195a60edc79d7159c60585c2dcdf96: Status 404 returned error can't find the container with id b40d0281b031cb243624f66011fdcac6bf195a60edc79d7159c60585c2dcdf96 Sep 18 06:46:49.067999 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:49.067978 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-87d618-5d94f66f4c-kwt2c"] Sep 18 06:46:49.075488 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:46:49.075462 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod584f37b8_a704_4ba3_bba0_1035c9365bda.slice/crio-3cbfdf3aef1979f1c86e16a4c49b9a6c7cac0bb3d699eabb2fbcb62ea6a3f51a WatchSource:0}: Error finding container 3cbfdf3aef1979f1c86e16a4c49b9a6c7cac0bb3d699eabb2fbcb62ea6a3f51a: Status 404 returned error can't find the container with id 3cbfdf3aef1979f1c86e16a4c49b9a6c7cac0bb3d699eabb2fbcb62ea6a3f51a Sep 18 06:46:49.943772 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:49.943740 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-87d618-5d94f66f4c-kwt2c" event={"ID":"584f37b8-a704-4ba3-bba0-1035c9365bda","Type":"ContainerStarted","Data":"c6813fb3b825e1b2bed599082a59983c8903a5331bee99e4b6ebf1ea6f75f6f1"} Sep 18 06:46:49.944210 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:49.943795 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-87d618-5d94f66f4c-kwt2c" event={"ID":"584f37b8-a704-4ba3-bba0-1035c9365bda","Type":"ContainerStarted","Data":"3cbfdf3aef1979f1c86e16a4c49b9a6c7cac0bb3d699eabb2fbcb62ea6a3f51a"} Sep 18 06:46:49.944210 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:49.943823 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-87d618-5d94f66f4c-kwt2c" Sep 18 06:46:49.945553 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:49.945525 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-87d618-6c8d8db6bb-m75bv" event={"ID":"b318b9a5-4a0a-4b3b-aa75-bc7879cebd45","Type":"ContainerStarted","Data":"e3e3f0eacb4d2a71b9a28e6533699d56cc7eb51791dcb650fc11f786c27620c7"} Sep 18 06:46:49.945648 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:49.945569 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-87d618-6c8d8db6bb-m75bv" event={"ID":"b318b9a5-4a0a-4b3b-aa75-bc7879cebd45","Type":"ContainerStarted","Data":"b40d0281b031cb243624f66011fdcac6bf195a60edc79d7159c60585c2dcdf96"} Sep 18 06:46:49.945648 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:49.945590 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-87d618-6c8d8db6bb-m75bv" Sep 18 06:46:49.962317 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:49.962265 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-87d618-5d94f66f4c-kwt2c" podStartSLOduration=1.962244954 podStartE2EDuration="1.962244954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:46:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:46:49.959697971 +0000 UTC m=+1471.699114168" watchObservedRunningTime="2026-09-18 06:46:49.962244954 +0000 UTC m=+1471.701661130" Sep 18 06:46:49.978524 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:49.978481 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-87d618-6c8d8db6bb-m75bv" podStartSLOduration=1.978468187 podStartE2EDuration="1.978468187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:46:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:46:49.975844894 +0000 UTC m=+1471.715261089" watchObservedRunningTime="2026-09-18 06:46:49.978468187 +0000 UTC m=+1471.717884361" Sep 18 06:46:54.356427 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:54.356393 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6"] Sep 18 06:46:54.356844 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:54.356644 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6" podUID="505d2a4d-d115-4775-9f00-85b367cebf79" containerName="istio-proxy" containerID="cri-o://eb732a675c9d0769b704e29d08654af003c5db59d0a6ed55d4209a54ba32ca11" gracePeriod=30 Sep 18 06:46:54.856696 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:54.856662 2567 patch_prober.go:28] interesting pod/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 18 06:46:54.856851 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:54.856729 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6" podUID="505d2a4d-d115-4775-9f00-85b367cebf79" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 18 06:46:55.046864 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:55.046831 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9"] Sep 18 06:46:55.051424 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:55.051400 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9" Sep 18 06:46:55.053803 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:55.053780 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-2f5ffe-openshift-default-dockercfg-kkvlf\"" Sep 18 06:46:55.060800 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:55.060712 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9"] Sep 18 06:46:55.137833 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:55.137764 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc649003-f227-419f-8036-612548cba528-workload-certs\") pod \"e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9\" (UID: \"dc649003-f227-419f-8036-612548cba528\") " pod="openshift-ingress/e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9" Sep 18 06:46:55.137833 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:55.137807 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc649003-f227-419f-8036-612548cba528-istio-podinfo\") pod \"e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9\" (UID: \"dc649003-f227-419f-8036-612548cba528\") " pod="openshift-ingress/e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9" Sep 18 06:46:55.137833 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:55.137827 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc649003-f227-419f-8036-612548cba528-istio-token\") pod \"e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9\" (UID: \"dc649003-f227-419f-8036-612548cba528\") " pod="openshift-ingress/e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9" Sep 18 06:46:55.138094 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:55.137936 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc649003-f227-419f-8036-612548cba528-istio-data\") pod \"e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9\" (UID: \"dc649003-f227-419f-8036-612548cba528\") " pod="openshift-ingress/e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9" Sep 18 06:46:55.138094 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:55.137981 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc649003-f227-419f-8036-612548cba528-workload-socket\") pod \"e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9\" (UID: \"dc649003-f227-419f-8036-612548cba528\") " pod="openshift-ingress/e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9" Sep 18 06:46:55.138094 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:55.138019 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc649003-f227-419f-8036-612548cba528-istio-envoy\") pod \"e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9\" (UID: \"dc649003-f227-419f-8036-612548cba528\") " pod="openshift-ingress/e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9" Sep 18 06:46:55.138094 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:55.138073 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc649003-f227-419f-8036-612548cba528-istiod-ca-cert\") pod \"e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9\" (UID: \"dc649003-f227-419f-8036-612548cba528\") " pod="openshift-ingress/e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9" Sep 18 06:46:55.138291 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:55.138167 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc649003-f227-419f-8036-612548cba528-credential-socket\") pod \"e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9\" (UID: \"dc649003-f227-419f-8036-612548cba528\") " pod="openshift-ingress/e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9" Sep 18 06:46:55.138291 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:55.138226 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x7fj\" (UniqueName: \"kubernetes.io/projected/dc649003-f227-419f-8036-612548cba528-kube-api-access-7x7fj\") pod \"e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9\" (UID: \"dc649003-f227-419f-8036-612548cba528\") " pod="openshift-ingress/e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9" Sep 18 06:46:55.239921 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:55.239876 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc649003-f227-419f-8036-612548cba528-credential-socket\") pod \"e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9\" (UID: \"dc649003-f227-419f-8036-612548cba528\") " pod="openshift-ingress/e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9" Sep 18 06:46:55.240510 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:55.240484 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x7fj\" (UniqueName: \"kubernetes.io/projected/dc649003-f227-419f-8036-612548cba528-kube-api-access-7x7fj\") pod \"e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9\" (UID: \"dc649003-f227-419f-8036-612548cba528\") " pod="openshift-ingress/e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9" Sep 18 06:46:55.240642 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:55.240381 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc649003-f227-419f-8036-612548cba528-credential-socket\") pod \"e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9\" (UID: \"dc649003-f227-419f-8036-612548cba528\") " pod="openshift-ingress/e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9" Sep 18 06:46:55.240642 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:55.240576 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc649003-f227-419f-8036-612548cba528-workload-certs\") pod \"e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9\" (UID: \"dc649003-f227-419f-8036-612548cba528\") " pod="openshift-ingress/e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9" Sep 18 06:46:55.240995 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:55.240955 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc649003-f227-419f-8036-612548cba528-workload-certs\") pod \"e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9\" (UID: \"dc649003-f227-419f-8036-612548cba528\") " pod="openshift-ingress/e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9" Sep 18 06:46:55.241110 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:55.241022 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc649003-f227-419f-8036-612548cba528-istio-podinfo\") pod \"e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9\" (UID: \"dc649003-f227-419f-8036-612548cba528\") " pod="openshift-ingress/e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9" Sep 18 06:46:55.241110 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:55.241076 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc649003-f227-419f-8036-612548cba528-istio-token\") pod \"e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9\" (UID: \"dc649003-f227-419f-8036-612548cba528\") " pod="openshift-ingress/e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9" Sep 18 06:46:55.241242 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:55.241155 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc649003-f227-419f-8036-612548cba528-istio-data\") pod \"e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9\" (UID: \"dc649003-f227-419f-8036-612548cba528\") " pod="openshift-ingress/e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9" Sep 18 06:46:55.241242 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:55.241183 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc649003-f227-419f-8036-612548cba528-workload-socket\") pod \"e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9\" (UID: \"dc649003-f227-419f-8036-612548cba528\") " pod="openshift-ingress/e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9" Sep 18 06:46:55.241242 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:55.241227 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc649003-f227-419f-8036-612548cba528-istio-envoy\") pod \"e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9\" (UID: \"dc649003-f227-419f-8036-612548cba528\") " pod="openshift-ingress/e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9" Sep 18 06:46:55.241420 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:55.241292 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc649003-f227-419f-8036-612548cba528-istiod-ca-cert\") pod \"e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9\" (UID: \"dc649003-f227-419f-8036-612548cba528\") " pod="openshift-ingress/e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9" Sep 18 06:46:55.241667 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:55.241640 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc649003-f227-419f-8036-612548cba528-workload-socket\") pod \"e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9\" (UID: \"dc649003-f227-419f-8036-612548cba528\") " pod="openshift-ingress/e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9" Sep 18 06:46:55.241780 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:55.241709 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc649003-f227-419f-8036-612548cba528-istio-data\") pod \"e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9\" (UID: \"dc649003-f227-419f-8036-612548cba528\") " pod="openshift-ingress/e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9" Sep 18 06:46:55.242112 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:55.242081 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc649003-f227-419f-8036-612548cba528-istiod-ca-cert\") pod \"e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9\" (UID: \"dc649003-f227-419f-8036-612548cba528\") " pod="openshift-ingress/e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9" Sep 18 06:46:55.243652 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:55.243626 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc649003-f227-419f-8036-612548cba528-istio-envoy\") pod \"e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9\" (UID: \"dc649003-f227-419f-8036-612548cba528\") " pod="openshift-ingress/e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9" Sep 18 06:46:55.243822 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:55.243801 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc649003-f227-419f-8036-612548cba528-istio-podinfo\") pod \"e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9\" (UID: \"dc649003-f227-419f-8036-612548cba528\") " pod="openshift-ingress/e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9" Sep 18 06:46:55.247688 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:55.247660 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x7fj\" (UniqueName: \"kubernetes.io/projected/dc649003-f227-419f-8036-612548cba528-kube-api-access-7x7fj\") pod \"e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9\" (UID: \"dc649003-f227-419f-8036-612548cba528\") " pod="openshift-ingress/e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9" Sep 18 06:46:55.248153 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:55.248133 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc649003-f227-419f-8036-612548cba528-istio-token\") pod \"e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9\" (UID: \"dc649003-f227-419f-8036-612548cba528\") " pod="openshift-ingress/e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9" Sep 18 06:46:55.366198 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:55.366167 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9" Sep 18 06:46:55.642324 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:55.642296 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9"] Sep 18 06:46:55.643142 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:46:55.643118 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc649003_f227_419f_8036_612548cba528.slice/crio-047211fc699fefea9d687a934f8b10b354ca6fff089c376e6aeb78ce2e69f435 WatchSource:0}: Error finding container 047211fc699fefea9d687a934f8b10b354ca6fff089c376e6aeb78ce2e69f435: Status 404 returned error can't find the container with id 047211fc699fefea9d687a934f8b10b354ca6fff089c376e6aeb78ce2e69f435 Sep 18 06:46:55.645609 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:55.645576 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:46:55.645726 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:55.645652 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:46:55.645726 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:55.645695 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:46:55.988146 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:55.988053 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9" event={"ID":"dc649003-f227-419f-8036-612548cba528","Type":"ContainerStarted","Data":"dae78cc7c8581d5fb3742253f6888cee3294bf63f1680bdd72e427f8a696a09a"} Sep 18 06:46:55.988146 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:55.988118 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9" event={"ID":"dc649003-f227-419f-8036-612548cba528","Type":"ContainerStarted","Data":"047211fc699fefea9d687a934f8b10b354ca6fff089c376e6aeb78ce2e69f435"} Sep 18 06:46:56.008997 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:56.008949 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9" podStartSLOduration=1.008934496 podStartE2EDuration="1.008934496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:46:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:46:56.006424978 +0000 UTC m=+1477.745841180" watchObservedRunningTime="2026-09-18 06:46:56.008934496 +0000 UTC m=+1477.748350670" Sep 18 06:46:56.366324 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:56.366298 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9" Sep 18 06:46:56.371533 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:56.371512 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9" Sep 18 06:46:56.995315 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:56.995288 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9" Sep 18 06:46:56.996335 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:56.996315 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9" Sep 18 06:46:59.619579 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:59.619528 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6" Sep 18 06:46:59.686741 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:59.686716 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/505d2a4d-d115-4775-9f00-85b367cebf79-credential-socket\") pod \"505d2a4d-d115-4775-9f00-85b367cebf79\" (UID: \"505d2a4d-d115-4775-9f00-85b367cebf79\") " Sep 18 06:46:59.686892 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:59.686749 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/505d2a4d-d115-4775-9f00-85b367cebf79-istio-data\") pod \"505d2a4d-d115-4775-9f00-85b367cebf79\" (UID: \"505d2a4d-d115-4775-9f00-85b367cebf79\") " Sep 18 06:46:59.686892 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:59.686773 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/505d2a4d-d115-4775-9f00-85b367cebf79-workload-certs\") pod \"505d2a4d-d115-4775-9f00-85b367cebf79\" (UID: \"505d2a4d-d115-4775-9f00-85b367cebf79\") " Sep 18 06:46:59.686892 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:59.686874 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/505d2a4d-d115-4775-9f00-85b367cebf79-istio-podinfo\") pod \"505d2a4d-d115-4775-9f00-85b367cebf79\" (UID: \"505d2a4d-d115-4775-9f00-85b367cebf79\") " Sep 18 06:46:59.687075 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:59.686900 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/505d2a4d-d115-4775-9f00-85b367cebf79-istio-token\") pod \"505d2a4d-d115-4775-9f00-85b367cebf79\" (UID: \"505d2a4d-d115-4775-9f00-85b367cebf79\") " Sep 18 06:46:59.687075 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:59.686923 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/505d2a4d-d115-4775-9f00-85b367cebf79-istio-envoy\") pod \"505d2a4d-d115-4775-9f00-85b367cebf79\" (UID: \"505d2a4d-d115-4775-9f00-85b367cebf79\") " Sep 18 06:46:59.687075 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:59.686980 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sbwcv\" (UniqueName: \"kubernetes.io/projected/505d2a4d-d115-4775-9f00-85b367cebf79-kube-api-access-sbwcv\") pod \"505d2a4d-d115-4775-9f00-85b367cebf79\" (UID: \"505d2a4d-d115-4775-9f00-85b367cebf79\") " Sep 18 06:46:59.687075 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:59.687009 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/505d2a4d-d115-4775-9f00-85b367cebf79-istiod-ca-cert\") pod \"505d2a4d-d115-4775-9f00-85b367cebf79\" (UID: \"505d2a4d-d115-4775-9f00-85b367cebf79\") " Sep 18 06:46:59.687075 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:59.687006 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/505d2a4d-d115-4775-9f00-85b367cebf79-istio-data" (OuterVolumeSpecName: "istio-data") pod "505d2a4d-d115-4775-9f00-85b367cebf79" (UID: "505d2a4d-d115-4775-9f00-85b367cebf79"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:46:59.687075 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:59.687038 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/505d2a4d-d115-4775-9f00-85b367cebf79-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "505d2a4d-d115-4775-9f00-85b367cebf79" (UID: "505d2a4d-d115-4775-9f00-85b367cebf79"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:46:59.687075 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:59.687042 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/505d2a4d-d115-4775-9f00-85b367cebf79-workload-socket\") pod \"505d2a4d-d115-4775-9f00-85b367cebf79\" (UID: \"505d2a4d-d115-4775-9f00-85b367cebf79\") " Sep 18 06:46:59.687481 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:59.687096 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/505d2a4d-d115-4775-9f00-85b367cebf79-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "505d2a4d-d115-4775-9f00-85b367cebf79" (UID: "505d2a4d-d115-4775-9f00-85b367cebf79"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:46:59.687481 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:59.687238 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/505d2a4d-d115-4775-9f00-85b367cebf79-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "505d2a4d-d115-4775-9f00-85b367cebf79" (UID: "505d2a4d-d115-4775-9f00-85b367cebf79"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:46:59.687481 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:59.687466 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/505d2a4d-d115-4775-9f00-85b367cebf79-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "505d2a4d-d115-4775-9f00-85b367cebf79" (UID: "505d2a4d-d115-4775-9f00-85b367cebf79"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 06:46:59.687642 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:59.687527 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/505d2a4d-d115-4775-9f00-85b367cebf79-workload-socket\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:46:59.687642 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:59.687547 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/505d2a4d-d115-4775-9f00-85b367cebf79-credential-socket\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:46:59.687642 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:59.687560 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/505d2a4d-d115-4775-9f00-85b367cebf79-istio-data\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:46:59.687642 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:59.687572 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/505d2a4d-d115-4775-9f00-85b367cebf79-workload-certs\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:46:59.689193 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:59.689169 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/505d2a4d-d115-4775-9f00-85b367cebf79-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "505d2a4d-d115-4775-9f00-85b367cebf79" (UID: "505d2a4d-d115-4775-9f00-85b367cebf79"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 06:46:59.689193 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:59.689175 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/505d2a4d-d115-4775-9f00-85b367cebf79-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "505d2a4d-d115-4775-9f00-85b367cebf79" (UID: "505d2a4d-d115-4775-9f00-85b367cebf79"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:46:59.689520 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:59.689492 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/505d2a4d-d115-4775-9f00-85b367cebf79-kube-api-access-sbwcv" (OuterVolumeSpecName: "kube-api-access-sbwcv") pod "505d2a4d-d115-4775-9f00-85b367cebf79" (UID: "505d2a4d-d115-4775-9f00-85b367cebf79"). InnerVolumeSpecName "kube-api-access-sbwcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 06:46:59.689520 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:59.689499 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/505d2a4d-d115-4775-9f00-85b367cebf79-istio-token" (OuterVolumeSpecName: "istio-token") pod "505d2a4d-d115-4775-9f00-85b367cebf79" (UID: "505d2a4d-d115-4775-9f00-85b367cebf79"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 06:46:59.788989 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:59.788947 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/505d2a4d-d115-4775-9f00-85b367cebf79-istio-podinfo\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:46:59.788989 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:59.788983 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/505d2a4d-d115-4775-9f00-85b367cebf79-istio-token\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:46:59.788989 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:59.788992 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/505d2a4d-d115-4775-9f00-85b367cebf79-istio-envoy\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:46:59.788989 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:59.789001 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbwcv\" (UniqueName: \"kubernetes.io/projected/505d2a4d-d115-4775-9f00-85b367cebf79-kube-api-access-sbwcv\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:46:59.789256 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:46:59.789009 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/505d2a4d-d115-4775-9f00-85b367cebf79-istiod-ca-cert\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:47:00.016101 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:00.016017 2567 generic.go:356] "Generic (PLEG): container finished" podID="505d2a4d-d115-4775-9f00-85b367cebf79" containerID="eb732a675c9d0769b704e29d08654af003c5db59d0a6ed55d4209a54ba32ca11" exitCode=0 Sep 18 06:47:00.016101 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:00.016065 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6" Sep 18 06:47:00.016280 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:00.016098 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6" event={"ID":"505d2a4d-d115-4775-9f00-85b367cebf79","Type":"ContainerDied","Data":"eb732a675c9d0769b704e29d08654af003c5db59d0a6ed55d4209a54ba32ca11"} Sep 18 06:47:00.016280 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:00.016155 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6" event={"ID":"505d2a4d-d115-4775-9f00-85b367cebf79","Type":"ContainerDied","Data":"f642bab8483fb2833032fe3584c3ef936ef3af45cb3c64cd36a7cd28d81df912"} Sep 18 06:47:00.016280 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:00.016180 2567 scope.go:117] "RemoveContainer" containerID="eb732a675c9d0769b704e29d08654af003c5db59d0a6ed55d4209a54ba32ca11" Sep 18 06:47:00.026919 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:00.026904 2567 scope.go:117] "RemoveContainer" containerID="af993aee090be7459197b964646142f2c590557a330acc6e01edb110de2d5e1d" Sep 18 06:47:00.036243 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:00.036226 2567 scope.go:117] "RemoveContainer" containerID="eb732a675c9d0769b704e29d08654af003c5db59d0a6ed55d4209a54ba32ca11" Sep 18 06:47:00.036538 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:47:00.036519 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb732a675c9d0769b704e29d08654af003c5db59d0a6ed55d4209a54ba32ca11\": container with ID starting with eb732a675c9d0769b704e29d08654af003c5db59d0a6ed55d4209a54ba32ca11 not found: ID does not exist" containerID="eb732a675c9d0769b704e29d08654af003c5db59d0a6ed55d4209a54ba32ca11" Sep 18 06:47:00.036626 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:00.036550 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb732a675c9d0769b704e29d08654af003c5db59d0a6ed55d4209a54ba32ca11"} err="failed to get container status \"eb732a675c9d0769b704e29d08654af003c5db59d0a6ed55d4209a54ba32ca11\": rpc error: code = NotFound desc = could not find container \"eb732a675c9d0769b704e29d08654af003c5db59d0a6ed55d4209a54ba32ca11\": container with ID starting with eb732a675c9d0769b704e29d08654af003c5db59d0a6ed55d4209a54ba32ca11 not found: ID does not exist" Sep 18 06:47:00.036626 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:00.036574 2567 scope.go:117] "RemoveContainer" containerID="af993aee090be7459197b964646142f2c590557a330acc6e01edb110de2d5e1d" Sep 18 06:47:00.036835 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:47:00.036818 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af993aee090be7459197b964646142f2c590557a330acc6e01edb110de2d5e1d\": container with ID starting with af993aee090be7459197b964646142f2c590557a330acc6e01edb110de2d5e1d not found: ID does not exist" containerID="af993aee090be7459197b964646142f2c590557a330acc6e01edb110de2d5e1d" Sep 18 06:47:00.036888 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:00.036845 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af993aee090be7459197b964646142f2c590557a330acc6e01edb110de2d5e1d"} err="failed to get container status \"af993aee090be7459197b964646142f2c590557a330acc6e01edb110de2d5e1d\": rpc error: code = NotFound desc = could not find container \"af993aee090be7459197b964646142f2c590557a330acc6e01edb110de2d5e1d\": container with ID starting with af993aee090be7459197b964646142f2c590557a330acc6e01edb110de2d5e1d not found: ID does not exist" Sep 18 06:47:00.039288 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:00.039263 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6"] Sep 18 06:47:00.042116 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:00.042098 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-14ee59-openshift-default-77dc4b89fb-gkdr6"] Sep 18 06:47:00.446767 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:00.446738 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="505d2a4d-d115-4775-9f00-85b367cebf79" path="/var/lib/kubelet/pods/505d2a4d-d115-4775-9f00-85b367cebf79/volumes" Sep 18 06:47:00.956095 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:00.956065 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-87d618-6c8d8db6bb-m75bv" Sep 18 06:47:00.956579 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:00.956147 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-87d618-5d94f66f4c-kwt2c" Sep 18 06:47:20.507384 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:20.507344 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjr4h_eb059951-e263-40da-9155-4fd0cf242820/ovn-acl-logging/0.log" Sep 18 06:47:20.512742 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:20.512717 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjr4h_eb059951-e263-40da-9155-4fd0cf242820/ovn-acl-logging/0.log" Sep 18 06:47:20.546955 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:20.546932 2567 scope.go:117] "RemoveContainer" containerID="e10826b7926dc4af2fc1d6f23d547b3e6fb0815ebddb9e06ec9b2f89a8d8e20b" Sep 18 06:47:23.946867 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:23.946836 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-93683709-openshift-default-6df68648-ckchx"] Sep 18 06:47:23.947556 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:23.947538 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="505d2a4d-d115-4775-9f00-85b367cebf79" containerName="istio-proxy" Sep 18 06:47:23.947611 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:23.947560 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="505d2a4d-d115-4775-9f00-85b367cebf79" containerName="istio-proxy" Sep 18 06:47:23.947611 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:23.947597 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="505d2a4d-d115-4775-9f00-85b367cebf79" containerName="istio-proxy" Sep 18 06:47:23.947611 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:23.947607 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="505d2a4d-d115-4775-9f00-85b367cebf79" containerName="istio-proxy" Sep 18 06:47:23.947724 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:23.947712 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="505d2a4d-d115-4775-9f00-85b367cebf79" containerName="istio-proxy" Sep 18 06:47:23.952946 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:23.952927 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-93683709-openshift-default-6df68648-ckchx" Sep 18 06:47:23.956131 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:23.956110 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-93683709-openshift-default-dockercfg-l6cdd\"" Sep 18 06:47:23.961179 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:23.961158 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-93683709-openshift-default-6df68648-ckchx"] Sep 18 06:47:24.012098 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:24.012063 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bd7655f-4e2a-4d12-a54f-8d26ef8eb373-istio-data\") pod \"e2e-mt-93683709-openshift-default-6df68648-ckchx\" (UID: \"2bd7655f-4e2a-4d12-a54f-8d26ef8eb373\") " pod="openshift-ingress/e2e-mt-93683709-openshift-default-6df68648-ckchx" Sep 18 06:47:24.012098 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:24.012113 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bd7655f-4e2a-4d12-a54f-8d26ef8eb373-istiod-ca-cert\") pod \"e2e-mt-93683709-openshift-default-6df68648-ckchx\" (UID: \"2bd7655f-4e2a-4d12-a54f-8d26ef8eb373\") " pod="openshift-ingress/e2e-mt-93683709-openshift-default-6df68648-ckchx" Sep 18 06:47:24.012276 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:24.012147 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bd7655f-4e2a-4d12-a54f-8d26ef8eb373-istio-token\") pod \"e2e-mt-93683709-openshift-default-6df68648-ckchx\" (UID: \"2bd7655f-4e2a-4d12-a54f-8d26ef8eb373\") " pod="openshift-ingress/e2e-mt-93683709-openshift-default-6df68648-ckchx" Sep 18 06:47:24.012276 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:24.012228 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bd7655f-4e2a-4d12-a54f-8d26ef8eb373-workload-socket\") pod \"e2e-mt-93683709-openshift-default-6df68648-ckchx\" (UID: \"2bd7655f-4e2a-4d12-a54f-8d26ef8eb373\") " pod="openshift-ingress/e2e-mt-93683709-openshift-default-6df68648-ckchx" Sep 18 06:47:24.012395 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:24.012278 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xttlz\" (UniqueName: \"kubernetes.io/projected/2bd7655f-4e2a-4d12-a54f-8d26ef8eb373-kube-api-access-xttlz\") pod \"e2e-mt-93683709-openshift-default-6df68648-ckchx\" (UID: \"2bd7655f-4e2a-4d12-a54f-8d26ef8eb373\") " pod="openshift-ingress/e2e-mt-93683709-openshift-default-6df68648-ckchx" Sep 18 06:47:24.012395 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:24.012325 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bd7655f-4e2a-4d12-a54f-8d26ef8eb373-istio-envoy\") pod \"e2e-mt-93683709-openshift-default-6df68648-ckchx\" (UID: \"2bd7655f-4e2a-4d12-a54f-8d26ef8eb373\") " pod="openshift-ingress/e2e-mt-93683709-openshift-default-6df68648-ckchx" Sep 18 06:47:24.012395 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:24.012350 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bd7655f-4e2a-4d12-a54f-8d26ef8eb373-credential-socket\") pod \"e2e-mt-93683709-openshift-default-6df68648-ckchx\" (UID: \"2bd7655f-4e2a-4d12-a54f-8d26ef8eb373\") " pod="openshift-ingress/e2e-mt-93683709-openshift-default-6df68648-ckchx" Sep 18 06:47:24.012517 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:24.012409 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bd7655f-4e2a-4d12-a54f-8d26ef8eb373-istio-podinfo\") pod \"e2e-mt-93683709-openshift-default-6df68648-ckchx\" (UID: \"2bd7655f-4e2a-4d12-a54f-8d26ef8eb373\") " pod="openshift-ingress/e2e-mt-93683709-openshift-default-6df68648-ckchx" Sep 18 06:47:24.012517 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:24.012426 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bd7655f-4e2a-4d12-a54f-8d26ef8eb373-workload-certs\") pod \"e2e-mt-93683709-openshift-default-6df68648-ckchx\" (UID: \"2bd7655f-4e2a-4d12-a54f-8d26ef8eb373\") " pod="openshift-ingress/e2e-mt-93683709-openshift-default-6df68648-ckchx" Sep 18 06:47:24.044440 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:24.044399 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7c4f4b-2cgp6"] Sep 18 06:47:24.045244 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:24.045222 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="505d2a4d-d115-4775-9f00-85b367cebf79" containerName="istio-proxy" Sep 18 06:47:24.049486 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:24.049468 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7c4f4b-2cgp6" Sep 18 06:47:24.052746 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:24.052725 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 18 06:47:24.053192 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:24.053173 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-88t7h\"" Sep 18 06:47:24.055623 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:24.055589 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7c4f4b-2cgp6"] Sep 18 06:47:24.113259 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:24.113224 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bd7655f-4e2a-4d12-a54f-8d26ef8eb373-workload-socket\") pod \"e2e-mt-93683709-openshift-default-6df68648-ckchx\" (UID: \"2bd7655f-4e2a-4d12-a54f-8d26ef8eb373\") " pod="openshift-ingress/e2e-mt-93683709-openshift-default-6df68648-ckchx" Sep 18 06:47:24.113472 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:24.113268 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xttlz\" (UniqueName: \"kubernetes.io/projected/2bd7655f-4e2a-4d12-a54f-8d26ef8eb373-kube-api-access-xttlz\") pod \"e2e-mt-93683709-openshift-default-6df68648-ckchx\" (UID: \"2bd7655f-4e2a-4d12-a54f-8d26ef8eb373\") " pod="openshift-ingress/e2e-mt-93683709-openshift-default-6df68648-ckchx" Sep 18 06:47:24.113472 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:24.113292 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bd7655f-4e2a-4d12-a54f-8d26ef8eb373-istio-envoy\") pod \"e2e-mt-93683709-openshift-default-6df68648-ckchx\" (UID: \"2bd7655f-4e2a-4d12-a54f-8d26ef8eb373\") " pod="openshift-ingress/e2e-mt-93683709-openshift-default-6df68648-ckchx" Sep 18 06:47:24.113472 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:24.113312 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bd7655f-4e2a-4d12-a54f-8d26ef8eb373-credential-socket\") pod \"e2e-mt-93683709-openshift-default-6df68648-ckchx\" (UID: \"2bd7655f-4e2a-4d12-a54f-8d26ef8eb373\") " pod="openshift-ingress/e2e-mt-93683709-openshift-default-6df68648-ckchx" Sep 18 06:47:24.113472 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:24.113342 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bd7655f-4e2a-4d12-a54f-8d26ef8eb373-istio-podinfo\") pod \"e2e-mt-93683709-openshift-default-6df68648-ckchx\" (UID: \"2bd7655f-4e2a-4d12-a54f-8d26ef8eb373\") " pod="openshift-ingress/e2e-mt-93683709-openshift-default-6df68648-ckchx" Sep 18 06:47:24.113472 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:24.113360 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bd7655f-4e2a-4d12-a54f-8d26ef8eb373-workload-certs\") pod \"e2e-mt-93683709-openshift-default-6df68648-ckchx\" (UID: \"2bd7655f-4e2a-4d12-a54f-8d26ef8eb373\") " pod="openshift-ingress/e2e-mt-93683709-openshift-default-6df68648-ckchx" Sep 18 06:47:24.113472 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:24.113420 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bd7655f-4e2a-4d12-a54f-8d26ef8eb373-istio-data\") pod \"e2e-mt-93683709-openshift-default-6df68648-ckchx\" (UID: \"2bd7655f-4e2a-4d12-a54f-8d26ef8eb373\") " pod="openshift-ingress/e2e-mt-93683709-openshift-default-6df68648-ckchx" Sep 18 06:47:24.113472 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:24.113460 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bd7655f-4e2a-4d12-a54f-8d26ef8eb373-istiod-ca-cert\") pod \"e2e-mt-93683709-openshift-default-6df68648-ckchx\" (UID: \"2bd7655f-4e2a-4d12-a54f-8d26ef8eb373\") " pod="openshift-ingress/e2e-mt-93683709-openshift-default-6df68648-ckchx" Sep 18 06:47:24.113879 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:24.113484 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aee367e2-ff94-48fb-a13d-8e6f1c01d6b3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7c4f4b-2cgp6\" (UID: \"aee367e2-ff94-48fb-a13d-8e6f1c01d6b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7c4f4b-2cgp6" Sep 18 06:47:24.113879 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:24.113540 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bd7655f-4e2a-4d12-a54f-8d26ef8eb373-istio-token\") pod \"e2e-mt-93683709-openshift-default-6df68648-ckchx\" (UID: \"2bd7655f-4e2a-4d12-a54f-8d26ef8eb373\") " pod="openshift-ingress/e2e-mt-93683709-openshift-default-6df68648-ckchx" Sep 18 06:47:24.113879 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:24.113814 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bd7655f-4e2a-4d12-a54f-8d26ef8eb373-credential-socket\") pod \"e2e-mt-93683709-openshift-default-6df68648-ckchx\" (UID: \"2bd7655f-4e2a-4d12-a54f-8d26ef8eb373\") " pod="openshift-ingress/e2e-mt-93683709-openshift-default-6df68648-ckchx" Sep 18 06:47:24.113879 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:24.113855 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bd7655f-4e2a-4d12-a54f-8d26ef8eb373-istio-data\") pod \"e2e-mt-93683709-openshift-default-6df68648-ckchx\" (UID: \"2bd7655f-4e2a-4d12-a54f-8d26ef8eb373\") " pod="openshift-ingress/e2e-mt-93683709-openshift-default-6df68648-ckchx" Sep 18 06:47:24.114093 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:24.113895 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bd7655f-4e2a-4d12-a54f-8d26ef8eb373-workload-certs\") pod \"e2e-mt-93683709-openshift-default-6df68648-ckchx\" (UID: \"2bd7655f-4e2a-4d12-a54f-8d26ef8eb373\") " pod="openshift-ingress/e2e-mt-93683709-openshift-default-6df68648-ckchx" Sep 18 06:47:24.114093 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:24.113960 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bd7655f-4e2a-4d12-a54f-8d26ef8eb373-workload-socket\") pod \"e2e-mt-93683709-openshift-default-6df68648-ckchx\" (UID: \"2bd7655f-4e2a-4d12-a54f-8d26ef8eb373\") " pod="openshift-ingress/e2e-mt-93683709-openshift-default-6df68648-ckchx" Sep 18 06:47:24.114197 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:24.114142 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bd7655f-4e2a-4d12-a54f-8d26ef8eb373-istiod-ca-cert\") pod \"e2e-mt-93683709-openshift-default-6df68648-ckchx\" (UID: \"2bd7655f-4e2a-4d12-a54f-8d26ef8eb373\") " pod="openshift-ingress/e2e-mt-93683709-openshift-default-6df68648-ckchx" Sep 18 06:47:24.115678 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:24.115643 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bd7655f-4e2a-4d12-a54f-8d26ef8eb373-istio-envoy\") pod \"e2e-mt-93683709-openshift-default-6df68648-ckchx\" (UID: \"2bd7655f-4e2a-4d12-a54f-8d26ef8eb373\") " pod="openshift-ingress/e2e-mt-93683709-openshift-default-6df68648-ckchx" Sep 18 06:47:24.115862 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:24.115846 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bd7655f-4e2a-4d12-a54f-8d26ef8eb373-istio-podinfo\") pod \"e2e-mt-93683709-openshift-default-6df68648-ckchx\" (UID: \"2bd7655f-4e2a-4d12-a54f-8d26ef8eb373\") " pod="openshift-ingress/e2e-mt-93683709-openshift-default-6df68648-ckchx" Sep 18 06:47:24.120899 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:24.120880 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bd7655f-4e2a-4d12-a54f-8d26ef8eb373-istio-token\") pod \"e2e-mt-93683709-openshift-default-6df68648-ckchx\" (UID: \"2bd7655f-4e2a-4d12-a54f-8d26ef8eb373\") " pod="openshift-ingress/e2e-mt-93683709-openshift-default-6df68648-ckchx" Sep 18 06:47:24.121353 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:24.121330 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xttlz\" (UniqueName: \"kubernetes.io/projected/2bd7655f-4e2a-4d12-a54f-8d26ef8eb373-kube-api-access-xttlz\") pod \"e2e-mt-93683709-openshift-default-6df68648-ckchx\" (UID: \"2bd7655f-4e2a-4d12-a54f-8d26ef8eb373\") " pod="openshift-ingress/e2e-mt-93683709-openshift-default-6df68648-ckchx" Sep 18 06:47:24.214339 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:24.214264 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aee367e2-ff94-48fb-a13d-8e6f1c01d6b3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7c4f4b-2cgp6\" (UID: \"aee367e2-ff94-48fb-a13d-8e6f1c01d6b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7c4f4b-2cgp6" Sep 18 06:47:24.214883 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:24.214858 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aee367e2-ff94-48fb-a13d-8e6f1c01d6b3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7c4f4b-2cgp6\" (UID: \"aee367e2-ff94-48fb-a13d-8e6f1c01d6b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7c4f4b-2cgp6" Sep 18 06:47:24.266687 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:24.266661 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-93683709-openshift-default-6df68648-ckchx" Sep 18 06:47:24.372887 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:24.372858 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7c4f4b-2cgp6" Sep 18 06:47:24.549626 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:47:24.549589 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bd7655f_4e2a_4d12_a54f_8d26ef8eb373.slice/crio-a35f195ebd4828dafb806e34b8173692bb75ed2fb28ca7c24cc831900e13340e WatchSource:0}: Error finding container a35f195ebd4828dafb806e34b8173692bb75ed2fb28ca7c24cc831900e13340e: Status 404 returned error can't find the container with id a35f195ebd4828dafb806e34b8173692bb75ed2fb28ca7c24cc831900e13340e Sep 18 06:47:24.551148 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:24.551124 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-93683709-openshift-default-6df68648-ckchx"] Sep 18 06:47:24.552342 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:24.552301 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:47:24.552440 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:24.552406 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:47:24.552478 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:24.552452 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:47:24.650075 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:24.650041 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7c4f4b-2cgp6"] Sep 18 06:47:24.651077 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:47:24.651049 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaee367e2_ff94_48fb_a13d_8e6f1c01d6b3.slice/crio-ded43a5e2be30ccd5744366eb0e4b6914ba53964d0e898a6030f9d8a13d821ea WatchSource:0}: Error finding container ded43a5e2be30ccd5744366eb0e4b6914ba53964d0e898a6030f9d8a13d821ea: Status 404 returned error can't find the container with id ded43a5e2be30ccd5744366eb0e4b6914ba53964d0e898a6030f9d8a13d821ea Sep 18 06:47:25.206981 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:25.206889 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7c4f4b-2cgp6" event={"ID":"aee367e2-ff94-48fb-a13d-8e6f1c01d6b3","Type":"ContainerStarted","Data":"11fb717c85d6d7518f03bd27f0e9dcaa06e32d96ea1199a87a5a35288a91b002"} Sep 18 06:47:25.206981 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:25.206953 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7c4f4b-2cgp6" event={"ID":"aee367e2-ff94-48fb-a13d-8e6f1c01d6b3","Type":"ContainerStarted","Data":"ded43a5e2be30ccd5744366eb0e4b6914ba53964d0e898a6030f9d8a13d821ea"} Sep 18 06:47:25.208537 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:25.208514 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-93683709-openshift-default-6df68648-ckchx" event={"ID":"2bd7655f-4e2a-4d12-a54f-8d26ef8eb373","Type":"ContainerStarted","Data":"dd8532918004f2e04ec69a11fcc99ccf99c22a2f7b08ce4e2bfc5a85b1eb91b8"} Sep 18 06:47:25.208652 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:25.208558 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-93683709-openshift-default-6df68648-ckchx" event={"ID":"2bd7655f-4e2a-4d12-a54f-8d26ef8eb373","Type":"ContainerStarted","Data":"a35f195ebd4828dafb806e34b8173692bb75ed2fb28ca7c24cc831900e13340e"} Sep 18 06:47:25.223403 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:25.223342 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7c4f4b-2cgp6" podStartSLOduration=1.22332759 podStartE2EDuration="1.22332759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:47:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:47:25.220673949 +0000 UTC m=+1506.960090144" watchObservedRunningTime="2026-09-18 06:47:25.22332759 +0000 UTC m=+1506.962743764" Sep 18 06:47:25.239807 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:25.239765 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-93683709-openshift-default-6df68648-ckchx" podStartSLOduration=2.239751745 podStartE2EDuration="2.239751745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:47:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:47:25.237246507 +0000 UTC m=+1506.976662720" watchObservedRunningTime="2026-09-18 06:47:25.239751745 +0000 UTC m=+1506.979167918" Sep 18 06:47:25.267594 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:25.267568 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-93683709-openshift-default-6df68648-ckchx" Sep 18 06:47:25.272644 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:25.272623 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-93683709-openshift-default-6df68648-ckchx" Sep 18 06:47:26.216577 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:26.216543 2567 generic.go:356] "Generic (PLEG): container finished" podID="aee367e2-ff94-48fb-a13d-8e6f1c01d6b3" containerID="11fb717c85d6d7518f03bd27f0e9dcaa06e32d96ea1199a87a5a35288a91b002" exitCode=0 Sep 18 06:47:26.216961 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:26.216625 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7c4f4b-2cgp6" event={"ID":"aee367e2-ff94-48fb-a13d-8e6f1c01d6b3","Type":"ContainerDied","Data":"11fb717c85d6d7518f03bd27f0e9dcaa06e32d96ea1199a87a5a35288a91b002"} Sep 18 06:47:26.216961 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:26.216903 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-93683709-openshift-default-6df68648-ckchx" Sep 18 06:47:26.217676 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:26.217659 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-93683709-openshift-default-6df68648-ckchx" Sep 18 06:47:27.365921 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:27.365897 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7c4f4b-2cgp6" Sep 18 06:47:27.445825 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:27.445797 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aee367e2-ff94-48fb-a13d-8e6f1c01d6b3-maas-api-service-ca\") pod \"aee367e2-ff94-48fb-a13d-8e6f1c01d6b3\" (UID: \"aee367e2-ff94-48fb-a13d-8e6f1c01d6b3\") " Sep 18 06:47:27.446103 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:27.446081 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aee367e2-ff94-48fb-a13d-8e6f1c01d6b3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "aee367e2-ff94-48fb-a13d-8e6f1c01d6b3" (UID: "aee367e2-ff94-48fb-a13d-8e6f1c01d6b3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 06:47:27.546655 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:27.546575 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aee367e2-ff94-48fb-a13d-8e6f1c01d6b3-maas-api-service-ca\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:47:28.231675 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:28.231644 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7c4f4b-2cgp6" Sep 18 06:47:28.231863 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:28.231681 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7c4f4b-2cgp6" event={"ID":"aee367e2-ff94-48fb-a13d-8e6f1c01d6b3","Type":"ContainerDied","Data":"ded43a5e2be30ccd5744366eb0e4b6914ba53964d0e898a6030f9d8a13d821ea"} Sep 18 06:47:28.231863 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:28.231733 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ded43a5e2be30ccd5744366eb0e4b6914ba53964d0e898a6030f9d8a13d821ea" Sep 18 06:47:34.938664 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:34.938591 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7c4f4b-2cgp6"] Sep 18 06:47:34.941410 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:34.941388 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7c4f4b-2cgp6"] Sep 18 06:47:36.446275 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:36.446240 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aee367e2-ff94-48fb-a13d-8e6f1c01d6b3" path="/var/lib/kubelet/pods/aee367e2-ff94-48fb-a13d-8e6f1c01d6b3/volumes" Sep 18 06:47:41.302934 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:41.302899 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw"] Sep 18 06:47:41.303413 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:41.303204 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw" podUID="504bef64-f0e4-4856-a251-9296702689c4" containerName="istio-proxy" containerID="cri-o://06caf4dd97134a797990b3927b1a105051a8dda775fc8fe29cc4456cb6f943b0" gracePeriod=30 Sep 18 06:47:41.630560 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:41.630528 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-708b6f-wwdmn"] Sep 18 06:47:41.631085 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:41.631065 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aee367e2-ff94-48fb-a13d-8e6f1c01d6b3" containerName="revoke-keys" Sep 18 06:47:41.631190 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:41.631088 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="aee367e2-ff94-48fb-a13d-8e6f1c01d6b3" containerName="revoke-keys" Sep 18 06:47:41.631249 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:41.631188 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="aee367e2-ff94-48fb-a13d-8e6f1c01d6b3" containerName="revoke-keys" Sep 18 06:47:41.634605 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:41.634585 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-708b6f-wwdmn" Sep 18 06:47:41.637233 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:41.637208 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-88t7h\"" Sep 18 06:47:41.637339 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:41.637244 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 18 06:47:41.640124 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:41.640102 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-708b6f-wwdmn"] Sep 18 06:47:41.793491 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:41.793463 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2263b64-3f5d-4729-91e2-0975fa000742-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-708b6f-wwdmn\" (UID: \"e2263b64-3f5d-4729-91e2-0975fa000742\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-708b6f-wwdmn" Sep 18 06:47:41.894143 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:41.894062 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2263b64-3f5d-4729-91e2-0975fa000742-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-708b6f-wwdmn\" (UID: \"e2263b64-3f5d-4729-91e2-0975fa000742\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-708b6f-wwdmn" Sep 18 06:47:41.894696 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:41.894678 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2263b64-3f5d-4729-91e2-0975fa000742-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-708b6f-wwdmn\" (UID: \"e2263b64-3f5d-4729-91e2-0975fa000742\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-708b6f-wwdmn" Sep 18 06:47:41.947079 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:41.947059 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-708b6f-wwdmn" Sep 18 06:47:42.226147 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:42.226118 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-708b6f-wwdmn"] Sep 18 06:47:42.226931 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:47:42.226901 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2263b64_3f5d_4729_91e2_0975fa000742.slice/crio-45150ecd9991e451cec81736dcbeb1dcd89ef8d42d9706e38d1a586e7b96f9ba WatchSource:0}: Error finding container 45150ecd9991e451cec81736dcbeb1dcd89ef8d42d9706e38d1a586e7b96f9ba: Status 404 returned error can't find the container with id 45150ecd9991e451cec81736dcbeb1dcd89ef8d42d9706e38d1a586e7b96f9ba Sep 18 06:47:42.326277 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:42.326248 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-708b6f-wwdmn" event={"ID":"e2263b64-3f5d-4729-91e2-0975fa000742","Type":"ContainerStarted","Data":"45150ecd9991e451cec81736dcbeb1dcd89ef8d42d9706e38d1a586e7b96f9ba"} Sep 18 06:47:43.334023 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:43.333991 2567 generic.go:356] "Generic (PLEG): container finished" podID="e2263b64-3f5d-4729-91e2-0975fa000742" containerID="875828f6afe6cdaad0997dbe2f492b96dbc7706e0706eaa25504c1092715bb24" exitCode=0 Sep 18 06:47:43.334400 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:43.334049 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-708b6f-wwdmn" event={"ID":"e2263b64-3f5d-4729-91e2-0975fa000742","Type":"ContainerDied","Data":"875828f6afe6cdaad0997dbe2f492b96dbc7706e0706eaa25504c1092715bb24"} Sep 18 06:47:44.489286 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:44.489260 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-708b6f-wwdmn" Sep 18 06:47:44.620317 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:44.620237 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2263b64-3f5d-4729-91e2-0975fa000742-maas-api-service-ca\") pod \"e2263b64-3f5d-4729-91e2-0975fa000742\" (UID: \"e2263b64-3f5d-4729-91e2-0975fa000742\") " Sep 18 06:47:44.620633 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:44.620610 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2263b64-3f5d-4729-91e2-0975fa000742-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e2263b64-3f5d-4729-91e2-0975fa000742" (UID: "e2263b64-3f5d-4729-91e2-0975fa000742"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 06:47:44.721509 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:44.721483 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2263b64-3f5d-4729-91e2-0975fa000742-maas-api-service-ca\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:47:45.350547 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:45.350509 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-708b6f-wwdmn" event={"ID":"e2263b64-3f5d-4729-91e2-0975fa000742","Type":"ContainerDied","Data":"45150ecd9991e451cec81736dcbeb1dcd89ef8d42d9706e38d1a586e7b96f9ba"} Sep 18 06:47:45.350801 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:45.350774 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-708b6f-wwdmn" Sep 18 06:47:45.350927 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:45.350908 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45150ecd9991e451cec81736dcbeb1dcd89ef8d42d9706e38d1a586e7b96f9ba" Sep 18 06:47:46.605395 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:46.605358 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw" Sep 18 06:47:46.742796 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:46.742762 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/504bef64-f0e4-4856-a251-9296702689c4-workload-socket\") pod \"504bef64-f0e4-4856-a251-9296702689c4\" (UID: \"504bef64-f0e4-4856-a251-9296702689c4\") " Sep 18 06:47:46.742970 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:46.742828 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bw2r2\" (UniqueName: \"kubernetes.io/projected/504bef64-f0e4-4856-a251-9296702689c4-kube-api-access-bw2r2\") pod \"504bef64-f0e4-4856-a251-9296702689c4\" (UID: \"504bef64-f0e4-4856-a251-9296702689c4\") " Sep 18 06:47:46.742970 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:46.742852 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/504bef64-f0e4-4856-a251-9296702689c4-istio-podinfo\") pod \"504bef64-f0e4-4856-a251-9296702689c4\" (UID: \"504bef64-f0e4-4856-a251-9296702689c4\") " Sep 18 06:47:46.742970 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:46.742879 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/504bef64-f0e4-4856-a251-9296702689c4-istio-envoy\") pod \"504bef64-f0e4-4856-a251-9296702689c4\" (UID: \"504bef64-f0e4-4856-a251-9296702689c4\") " Sep 18 06:47:46.742970 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:46.742927 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/504bef64-f0e4-4856-a251-9296702689c4-workload-certs\") pod \"504bef64-f0e4-4856-a251-9296702689c4\" (UID: \"504bef64-f0e4-4856-a251-9296702689c4\") " Sep 18 06:47:46.742970 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:46.742958 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/504bef64-f0e4-4856-a251-9296702689c4-istio-data\") pod \"504bef64-f0e4-4856-a251-9296702689c4\" (UID: \"504bef64-f0e4-4856-a251-9296702689c4\") " Sep 18 06:47:46.743247 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:46.742985 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/504bef64-f0e4-4856-a251-9296702689c4-istio-token\") pod \"504bef64-f0e4-4856-a251-9296702689c4\" (UID: \"504bef64-f0e4-4856-a251-9296702689c4\") " Sep 18 06:47:46.743247 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:46.743000 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/504bef64-f0e4-4856-a251-9296702689c4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "504bef64-f0e4-4856-a251-9296702689c4" (UID: "504bef64-f0e4-4856-a251-9296702689c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:47:46.743247 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:46.743024 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/504bef64-f0e4-4856-a251-9296702689c4-istiod-ca-cert\") pod \"504bef64-f0e4-4856-a251-9296702689c4\" (UID: \"504bef64-f0e4-4856-a251-9296702689c4\") " Sep 18 06:47:46.743247 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:46.743127 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/504bef64-f0e4-4856-a251-9296702689c4-credential-socket\") pod \"504bef64-f0e4-4856-a251-9296702689c4\" (UID: \"504bef64-f0e4-4856-a251-9296702689c4\") " Sep 18 06:47:46.743247 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:46.743208 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/504bef64-f0e4-4856-a251-9296702689c4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "504bef64-f0e4-4856-a251-9296702689c4" (UID: "504bef64-f0e4-4856-a251-9296702689c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:47:46.743247 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:46.743225 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/504bef64-f0e4-4856-a251-9296702689c4-istio-data" (OuterVolumeSpecName: "istio-data") pod "504bef64-f0e4-4856-a251-9296702689c4" (UID: "504bef64-f0e4-4856-a251-9296702689c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:47:46.743554 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:46.743405 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/504bef64-f0e4-4856-a251-9296702689c4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "504bef64-f0e4-4856-a251-9296702689c4" (UID: "504bef64-f0e4-4856-a251-9296702689c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:47:46.743554 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:46.743478 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/504bef64-f0e4-4856-a251-9296702689c4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "504bef64-f0e4-4856-a251-9296702689c4" (UID: "504bef64-f0e4-4856-a251-9296702689c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 06:47:46.743667 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:46.743587 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/504bef64-f0e4-4856-a251-9296702689c4-credential-socket\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:47:46.743667 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:46.743608 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/504bef64-f0e4-4856-a251-9296702689c4-workload-socket\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:47:46.743667 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:46.743622 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/504bef64-f0e4-4856-a251-9296702689c4-workload-certs\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:47:46.743667 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:46.743635 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/504bef64-f0e4-4856-a251-9296702689c4-istio-data\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:47:46.743667 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:46.743648 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/504bef64-f0e4-4856-a251-9296702689c4-istiod-ca-cert\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:47:46.745212 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:46.745185 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/504bef64-f0e4-4856-a251-9296702689c4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "504bef64-f0e4-4856-a251-9296702689c4" (UID: "504bef64-f0e4-4856-a251-9296702689c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 06:47:46.745212 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:46.745185 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/504bef64-f0e4-4856-a251-9296702689c4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "504bef64-f0e4-4856-a251-9296702689c4" (UID: "504bef64-f0e4-4856-a251-9296702689c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:47:46.745359 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:46.745283 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/504bef64-f0e4-4856-a251-9296702689c4-kube-api-access-bw2r2" (OuterVolumeSpecName: "kube-api-access-bw2r2") pod "504bef64-f0e4-4856-a251-9296702689c4" (UID: "504bef64-f0e4-4856-a251-9296702689c4"). InnerVolumeSpecName "kube-api-access-bw2r2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 06:47:46.745359 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:46.745289 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/504bef64-f0e4-4856-a251-9296702689c4-istio-token" (OuterVolumeSpecName: "istio-token") pod "504bef64-f0e4-4856-a251-9296702689c4" (UID: "504bef64-f0e4-4856-a251-9296702689c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 06:47:46.844614 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:46.844540 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bw2r2\" (UniqueName: \"kubernetes.io/projected/504bef64-f0e4-4856-a251-9296702689c4-kube-api-access-bw2r2\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:47:46.844614 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:46.844569 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/504bef64-f0e4-4856-a251-9296702689c4-istio-podinfo\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:47:46.844614 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:46.844583 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/504bef64-f0e4-4856-a251-9296702689c4-istio-envoy\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:47:46.844614 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:46.844595 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/504bef64-f0e4-4856-a251-9296702689c4-istio-token\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:47:47.375695 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:47.375663 2567 generic.go:356] "Generic (PLEG): container finished" podID="504bef64-f0e4-4856-a251-9296702689c4" containerID="06caf4dd97134a797990b3927b1a105051a8dda775fc8fe29cc4456cb6f943b0" exitCode=0 Sep 18 06:47:47.375904 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:47.375724 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw" Sep 18 06:47:47.375904 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:47.375757 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw" event={"ID":"504bef64-f0e4-4856-a251-9296702689c4","Type":"ContainerDied","Data":"06caf4dd97134a797990b3927b1a105051a8dda775fc8fe29cc4456cb6f943b0"} Sep 18 06:47:47.375904 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:47.375796 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw" event={"ID":"504bef64-f0e4-4856-a251-9296702689c4","Type":"ContainerDied","Data":"214d3c91b99a6d8993a819e8586a8a81e7df7881248cdde89c899ac09810f596"} Sep 18 06:47:47.375904 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:47.375817 2567 scope.go:117] "RemoveContainer" containerID="06caf4dd97134a797990b3927b1a105051a8dda775fc8fe29cc4456cb6f943b0" Sep 18 06:47:47.387003 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:47.386985 2567 scope.go:117] "RemoveContainer" containerID="280d89efcbcbad5984a021c0fac52bcf28d945b59cb8c97b0494623e0cdc1835" Sep 18 06:47:47.396056 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:47.396036 2567 scope.go:117] "RemoveContainer" containerID="06caf4dd97134a797990b3927b1a105051a8dda775fc8fe29cc4456cb6f943b0" Sep 18 06:47:47.396318 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:47:47.396296 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06caf4dd97134a797990b3927b1a105051a8dda775fc8fe29cc4456cb6f943b0\": container with ID starting with 06caf4dd97134a797990b3927b1a105051a8dda775fc8fe29cc4456cb6f943b0 not found: ID does not exist" containerID="06caf4dd97134a797990b3927b1a105051a8dda775fc8fe29cc4456cb6f943b0" Sep 18 06:47:47.396402 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:47.396327 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06caf4dd97134a797990b3927b1a105051a8dda775fc8fe29cc4456cb6f943b0"} err="failed to get container status \"06caf4dd97134a797990b3927b1a105051a8dda775fc8fe29cc4456cb6f943b0\": rpc error: code = NotFound desc = could not find container \"06caf4dd97134a797990b3927b1a105051a8dda775fc8fe29cc4456cb6f943b0\": container with ID starting with 06caf4dd97134a797990b3927b1a105051a8dda775fc8fe29cc4456cb6f943b0 not found: ID does not exist" Sep 18 06:47:47.396402 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:47.396343 2567 scope.go:117] "RemoveContainer" containerID="280d89efcbcbad5984a021c0fac52bcf28d945b59cb8c97b0494623e0cdc1835" Sep 18 06:47:47.396617 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:47:47.396598 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"280d89efcbcbad5984a021c0fac52bcf28d945b59cb8c97b0494623e0cdc1835\": container with ID starting with 280d89efcbcbad5984a021c0fac52bcf28d945b59cb8c97b0494623e0cdc1835 not found: ID does not exist" containerID="280d89efcbcbad5984a021c0fac52bcf28d945b59cb8c97b0494623e0cdc1835" Sep 18 06:47:47.396659 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:47.396625 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"280d89efcbcbad5984a021c0fac52bcf28d945b59cb8c97b0494623e0cdc1835"} err="failed to get container status \"280d89efcbcbad5984a021c0fac52bcf28d945b59cb8c97b0494623e0cdc1835\": rpc error: code = NotFound desc = could not find container \"280d89efcbcbad5984a021c0fac52bcf28d945b59cb8c97b0494623e0cdc1835\": container with ID starting with 280d89efcbcbad5984a021c0fac52bcf28d945b59cb8c97b0494623e0cdc1835 not found: ID does not exist" Sep 18 06:47:47.401793 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:47.401742 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw"] Sep 18 06:47:47.405377 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:47.405347 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-7c4f4b-openshift-default-6d485db978-sn4vw"] Sep 18 06:47:48.445951 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:48.445919 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="504bef64-f0e4-4856-a251-9296702689c4" path="/var/lib/kubelet/pods/504bef64-f0e4-4856-a251-9296702689c4/volumes" Sep 18 06:47:52.334329 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:52.334300 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-708b6f-29828565-g4m2z"] Sep 18 06:47:52.337735 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:52.337709 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-708b6f-29828565-g4m2z"] Sep 18 06:47:52.362713 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:52.362683 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-708b6f-654cf9f9db-mfx72"] Sep 18 06:47:52.362957 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:52.362928 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-708b6f-654cf9f9db-mfx72" podUID="1caa877a-113d-4a99-a443-c9374528c9eb" containerName="payload-pre-processing" containerID="cri-o://d4f9d526112b4dc742db8c122609bfd485820cafea38c7e1d12b18e46fb83bb4" gracePeriod=30 Sep 18 06:47:52.448195 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:52.448157 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="263178e6-66a0-4b7c-8b94-d4337e921084" path="/var/lib/kubelet/pods/263178e6-66a0-4b7c-8b94-d4337e921084/volumes" Sep 18 06:47:52.509321 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:52.509291 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-708b6f-wwdmn"] Sep 18 06:47:52.512318 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:52.512295 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-708b6f-wwdmn"] Sep 18 06:47:52.665093 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:52.665063 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-708b6f-654cf9f9db-mfx72" Sep 18 06:47:52.718442 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:52.718411 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1caa877a-113d-4a99-a443-c9374528c9eb-plugins-config-volume\") pod \"1caa877a-113d-4a99-a443-c9374528c9eb\" (UID: \"1caa877a-113d-4a99-a443-c9374528c9eb\") " Sep 18 06:47:52.718442 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:52.718443 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjflc\" (UniqueName: \"kubernetes.io/projected/1caa877a-113d-4a99-a443-c9374528c9eb-kube-api-access-xjflc\") pod \"1caa877a-113d-4a99-a443-c9374528c9eb\" (UID: \"1caa877a-113d-4a99-a443-c9374528c9eb\") " Sep 18 06:47:52.718808 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:52.718786 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1caa877a-113d-4a99-a443-c9374528c9eb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1caa877a-113d-4a99-a443-c9374528c9eb" (UID: "1caa877a-113d-4a99-a443-c9374528c9eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 06:47:52.718926 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:52.718908 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1caa877a-113d-4a99-a443-c9374528c9eb-plugins-config-volume\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:47:52.720626 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:52.720606 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1caa877a-113d-4a99-a443-c9374528c9eb-kube-api-access-xjflc" (OuterVolumeSpecName: "kube-api-access-xjflc") pod "1caa877a-113d-4a99-a443-c9374528c9eb" (UID: "1caa877a-113d-4a99-a443-c9374528c9eb"). InnerVolumeSpecName "kube-api-access-xjflc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 06:47:52.819751 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:52.819717 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjflc\" (UniqueName: \"kubernetes.io/projected/1caa877a-113d-4a99-a443-c9374528c9eb-kube-api-access-xjflc\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:47:53.429415 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:53.429352 2567 generic.go:356] "Generic (PLEG): container finished" podID="1caa877a-113d-4a99-a443-c9374528c9eb" containerID="d4f9d526112b4dc742db8c122609bfd485820cafea38c7e1d12b18e46fb83bb4" exitCode=0 Sep 18 06:47:53.429844 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:53.429430 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-708b6f-654cf9f9db-mfx72" Sep 18 06:47:53.429844 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:53.429435 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-708b6f-654cf9f9db-mfx72" event={"ID":"1caa877a-113d-4a99-a443-c9374528c9eb","Type":"ContainerDied","Data":"d4f9d526112b4dc742db8c122609bfd485820cafea38c7e1d12b18e46fb83bb4"} Sep 18 06:47:53.429844 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:53.429493 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-708b6f-654cf9f9db-mfx72" event={"ID":"1caa877a-113d-4a99-a443-c9374528c9eb","Type":"ContainerDied","Data":"20ea43693829bf348a0282e2f9da93d6551e486b5b664f22b61e4384427ae14d"} Sep 18 06:47:53.429844 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:53.429518 2567 scope.go:117] "RemoveContainer" containerID="d4f9d526112b4dc742db8c122609bfd485820cafea38c7e1d12b18e46fb83bb4" Sep 18 06:47:53.439414 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:53.439397 2567 scope.go:117] "RemoveContainer" containerID="d4f9d526112b4dc742db8c122609bfd485820cafea38c7e1d12b18e46fb83bb4" Sep 18 06:47:53.439720 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:47:53.439701 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4f9d526112b4dc742db8c122609bfd485820cafea38c7e1d12b18e46fb83bb4\": container with ID starting with d4f9d526112b4dc742db8c122609bfd485820cafea38c7e1d12b18e46fb83bb4 not found: ID does not exist" containerID="d4f9d526112b4dc742db8c122609bfd485820cafea38c7e1d12b18e46fb83bb4" Sep 18 06:47:53.439790 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:53.439733 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4f9d526112b4dc742db8c122609bfd485820cafea38c7e1d12b18e46fb83bb4"} err="failed to get container status \"d4f9d526112b4dc742db8c122609bfd485820cafea38c7e1d12b18e46fb83bb4\": rpc error: code = NotFound desc = could not find container \"d4f9d526112b4dc742db8c122609bfd485820cafea38c7e1d12b18e46fb83bb4\": container with ID starting with d4f9d526112b4dc742db8c122609bfd485820cafea38c7e1d12b18e46fb83bb4 not found: ID does not exist" Sep 18 06:47:53.453957 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:53.453721 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-708b6f-654cf9f9db-mfx72"] Sep 18 06:47:53.463684 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:53.458939 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-708b6f-654cf9f9db-mfx72"] Sep 18 06:47:54.440978 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:54.440942 2567 generic.go:356] "Generic (PLEG): container finished" podID="2bd7655f-4e2a-4d12-a54f-8d26ef8eb373" containerID="dd8532918004f2e04ec69a11fcc99ccf99c22a2f7b08ce4e2bfc5a85b1eb91b8" exitCode=0 Sep 18 06:47:54.447107 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:54.447080 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1caa877a-113d-4a99-a443-c9374528c9eb" path="/var/lib/kubelet/pods/1caa877a-113d-4a99-a443-c9374528c9eb/volumes" Sep 18 06:47:54.447520 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:54.447505 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2263b64-3f5d-4729-91e2-0975fa000742" path="/var/lib/kubelet/pods/e2263b64-3f5d-4729-91e2-0975fa000742/volumes" Sep 18 06:47:54.447808 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:54.447789 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-93683709-openshift-default-6df68648-ckchx" event={"ID":"2bd7655f-4e2a-4d12-a54f-8d26ef8eb373","Type":"ContainerDied","Data":"dd8532918004f2e04ec69a11fcc99ccf99c22a2f7b08ce4e2bfc5a85b1eb91b8"} Sep 18 06:47:54.448164 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:54.448151 2567 scope.go:117] "RemoveContainer" containerID="dd8532918004f2e04ec69a11fcc99ccf99c22a2f7b08ce4e2bfc5a85b1eb91b8" Sep 18 06:47:54.449235 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:54.449201 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:47:54.449296 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:54.449280 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:47:54.449332 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:54.449311 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:47:55.454053 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:55.454011 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-93683709-openshift-default-6df68648-ckchx" event={"ID":"2bd7655f-4e2a-4d12-a54f-8d26ef8eb373","Type":"ContainerStarted","Data":"95c1557c709d03768a9a38d14d80a8ce5b50a292aacd161cf57ff4ee3e5a35ad"} Sep 18 06:47:56.217619 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:56.217589 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-93683709-openshift-default-6df68648-ckchx" Sep 18 06:47:56.268604 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:56.268569 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-93683709-openshift-default-6df68648-ckchx" Sep 18 06:47:56.268791 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:56.268704 2567 patch_prober.go:28] interesting pod/e2e-mt-93683709-openshift-default-6df68648-ckchx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Sep 18 06:47:56.268791 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:56.268748 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-93683709-openshift-default-6df68648-ckchx" podUID="2bd7655f-4e2a-4d12-a54f-8d26ef8eb373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Sep 18 06:47:57.268378 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:57.268318 2567 patch_prober.go:28] interesting pod/e2e-mt-93683709-openshift-default-6df68648-ckchx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Sep 18 06:47:57.268898 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:57.268412 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-93683709-openshift-default-6df68648-ckchx" podUID="2bd7655f-4e2a-4d12-a54f-8d26ef8eb373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Sep 18 06:47:58.267788 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:58.267754 2567 patch_prober.go:28] interesting pod/e2e-mt-93683709-openshift-default-6df68648-ckchx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Sep 18 06:47:58.268096 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:58.268065 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-93683709-openshift-default-6df68648-ckchx" podUID="2bd7655f-4e2a-4d12-a54f-8d26ef8eb373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Sep 18 06:47:59.267658 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:59.267618 2567 patch_prober.go:28] interesting pod/e2e-mt-93683709-openshift-default-6df68648-ckchx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Sep 18 06:47:59.268254 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:47:59.268219 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-93683709-openshift-default-6df68648-ckchx" podUID="2bd7655f-4e2a-4d12-a54f-8d26ef8eb373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Sep 18 06:48:00.267940 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:48:00.267893 2567 patch_prober.go:28] interesting pod/e2e-mt-93683709-openshift-default-6df68648-ckchx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Sep 18 06:48:00.268447 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:48:00.267979 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-93683709-openshift-default-6df68648-ckchx" podUID="2bd7655f-4e2a-4d12-a54f-8d26ef8eb373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Sep 18 06:48:01.271565 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:48:01.271526 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-93683709-openshift-default-6df68648-ckchx" Sep 18 06:48:01.272452 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:48:01.272434 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-93683709-openshift-default-6df68648-ckchx" Sep 18 06:48:20.585955 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:48:20.585927 2567 scope.go:117] "RemoveContainer" containerID="1563c54b6a271e2a107894cf9da42def38cd9fd9e34c45b5702c3d92a18ccfea" Sep 18 06:48:38.094335 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:48:38.094299 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9"] Sep 18 06:48:38.095292 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:48:38.095233 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9" podUID="dc649003-f227-419f-8036-612548cba528" containerName="istio-proxy" containerID="cri-o://dae78cc7c8581d5fb3742253f6888cee3294bf63f1680bdd72e427f8a696a09a" gracePeriod=30 Sep 18 06:48:41.998061 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:48:41.998021 2567 patch_prober.go:28] interesting pod/e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 18 06:48:41.998440 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:48:41.998090 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9" podUID="dc649003-f227-419f-8036-612548cba528" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 18 06:48:43.367494 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:48:43.367438 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9" Sep 18 06:48:43.399918 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:48:43.399892 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc649003-f227-419f-8036-612548cba528-istio-token\") pod \"dc649003-f227-419f-8036-612548cba528\" (UID: \"dc649003-f227-419f-8036-612548cba528\") " Sep 18 06:48:43.400120 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:48:43.399961 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc649003-f227-419f-8036-612548cba528-istio-podinfo\") pod \"dc649003-f227-419f-8036-612548cba528\" (UID: \"dc649003-f227-419f-8036-612548cba528\") " Sep 18 06:48:43.400120 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:48:43.400005 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc649003-f227-419f-8036-612548cba528-workload-certs\") pod \"dc649003-f227-419f-8036-612548cba528\" (UID: \"dc649003-f227-419f-8036-612548cba528\") " Sep 18 06:48:43.400120 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:48:43.400020 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc649003-f227-419f-8036-612548cba528-workload-socket\") pod \"dc649003-f227-419f-8036-612548cba528\" (UID: \"dc649003-f227-419f-8036-612548cba528\") " Sep 18 06:48:43.400120 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:48:43.400044 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc649003-f227-419f-8036-612548cba528-istio-envoy\") pod \"dc649003-f227-419f-8036-612548cba528\" (UID: \"dc649003-f227-419f-8036-612548cba528\") " Sep 18 06:48:43.400120 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:48:43.400098 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7x7fj\" (UniqueName: \"kubernetes.io/projected/dc649003-f227-419f-8036-612548cba528-kube-api-access-7x7fj\") pod \"dc649003-f227-419f-8036-612548cba528\" (UID: \"dc649003-f227-419f-8036-612548cba528\") " Sep 18 06:48:43.400724 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:48:43.400142 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc649003-f227-419f-8036-612548cba528-istiod-ca-cert\") pod \"dc649003-f227-419f-8036-612548cba528\" (UID: \"dc649003-f227-419f-8036-612548cba528\") " Sep 18 06:48:43.400724 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:48:43.400184 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc649003-f227-419f-8036-612548cba528-credential-socket\") pod \"dc649003-f227-419f-8036-612548cba528\" (UID: \"dc649003-f227-419f-8036-612548cba528\") " Sep 18 06:48:43.400724 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:48:43.400228 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc649003-f227-419f-8036-612548cba528-istio-data\") pod \"dc649003-f227-419f-8036-612548cba528\" (UID: \"dc649003-f227-419f-8036-612548cba528\") " Sep 18 06:48:43.400724 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:48:43.400300 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc649003-f227-419f-8036-612548cba528-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dc649003-f227-419f-8036-612548cba528" (UID: "dc649003-f227-419f-8036-612548cba528"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:48:43.400724 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:48:43.400330 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc649003-f227-419f-8036-612548cba528-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dc649003-f227-419f-8036-612548cba528" (UID: "dc649003-f227-419f-8036-612548cba528"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:48:43.400724 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:48:43.400537 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc649003-f227-419f-8036-612548cba528-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dc649003-f227-419f-8036-612548cba528" (UID: "dc649003-f227-419f-8036-612548cba528"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:48:43.400724 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:48:43.400596 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc649003-f227-419f-8036-612548cba528-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dc649003-f227-419f-8036-612548cba528" (UID: "dc649003-f227-419f-8036-612548cba528"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 06:48:43.400724 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:48:43.400709 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc649003-f227-419f-8036-612548cba528-istio-data" (OuterVolumeSpecName: "istio-data") pod "dc649003-f227-419f-8036-612548cba528" (UID: "dc649003-f227-419f-8036-612548cba528"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:48:43.400724 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:48:43.400713 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc649003-f227-419f-8036-612548cba528-workload-certs\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:48:43.400724 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:48:43.400736 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc649003-f227-419f-8036-612548cba528-workload-socket\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:48:43.401139 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:48:43.400752 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc649003-f227-419f-8036-612548cba528-istiod-ca-cert\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:48:43.401139 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:48:43.400767 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc649003-f227-419f-8036-612548cba528-credential-socket\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:48:43.402194 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:48:43.402171 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc649003-f227-419f-8036-612548cba528-istio-token" (OuterVolumeSpecName: "istio-token") pod "dc649003-f227-419f-8036-612548cba528" (UID: "dc649003-f227-419f-8036-612548cba528"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 06:48:43.402642 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:48:43.402618 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc649003-f227-419f-8036-612548cba528-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dc649003-f227-419f-8036-612548cba528" (UID: "dc649003-f227-419f-8036-612548cba528"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:48:43.402750 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:48:43.402707 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc649003-f227-419f-8036-612548cba528-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dc649003-f227-419f-8036-612548cba528" (UID: "dc649003-f227-419f-8036-612548cba528"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 06:48:43.402802 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:48:43.402771 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc649003-f227-419f-8036-612548cba528-kube-api-access-7x7fj" (OuterVolumeSpecName: "kube-api-access-7x7fj") pod "dc649003-f227-419f-8036-612548cba528" (UID: "dc649003-f227-419f-8036-612548cba528"). InnerVolumeSpecName "kube-api-access-7x7fj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 06:48:43.501562 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:48:43.501529 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x7fj\" (UniqueName: \"kubernetes.io/projected/dc649003-f227-419f-8036-612548cba528-kube-api-access-7x7fj\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:48:43.501562 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:48:43.501559 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc649003-f227-419f-8036-612548cba528-istio-data\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:48:43.501697 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:48:43.501569 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc649003-f227-419f-8036-612548cba528-istio-token\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:48:43.501697 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:48:43.501578 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc649003-f227-419f-8036-612548cba528-istio-podinfo\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:48:43.501697 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:48:43.501587 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc649003-f227-419f-8036-612548cba528-istio-envoy\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:48:43.777985 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:48:43.777897 2567 generic.go:356] "Generic (PLEG): container finished" podID="dc649003-f227-419f-8036-612548cba528" containerID="dae78cc7c8581d5fb3742253f6888cee3294bf63f1680bdd72e427f8a696a09a" exitCode=0 Sep 18 06:48:43.778130 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:48:43.778071 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9" event={"ID":"dc649003-f227-419f-8036-612548cba528","Type":"ContainerDied","Data":"dae78cc7c8581d5fb3742253f6888cee3294bf63f1680bdd72e427f8a696a09a"} Sep 18 06:48:43.778190 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:48:43.778174 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9" event={"ID":"dc649003-f227-419f-8036-612548cba528","Type":"ContainerDied","Data":"047211fc699fefea9d687a934f8b10b354ca6fff089c376e6aeb78ce2e69f435"} Sep 18 06:48:43.778253 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:48:43.778216 2567 scope.go:117] "RemoveContainer" containerID="dae78cc7c8581d5fb3742253f6888cee3294bf63f1680bdd72e427f8a696a09a" Sep 18 06:48:43.779039 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:48:43.778451 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9" Sep 18 06:48:43.794511 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:48:43.794489 2567 scope.go:117] "RemoveContainer" containerID="dae78cc7c8581d5fb3742253f6888cee3294bf63f1680bdd72e427f8a696a09a" Sep 18 06:48:43.794778 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:48:43.794759 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dae78cc7c8581d5fb3742253f6888cee3294bf63f1680bdd72e427f8a696a09a\": container with ID starting with dae78cc7c8581d5fb3742253f6888cee3294bf63f1680bdd72e427f8a696a09a not found: ID does not exist" containerID="dae78cc7c8581d5fb3742253f6888cee3294bf63f1680bdd72e427f8a696a09a" Sep 18 06:48:43.794859 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:48:43.794790 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dae78cc7c8581d5fb3742253f6888cee3294bf63f1680bdd72e427f8a696a09a"} err="failed to get container status \"dae78cc7c8581d5fb3742253f6888cee3294bf63f1680bdd72e427f8a696a09a\": rpc error: code = NotFound desc = could not find container \"dae78cc7c8581d5fb3742253f6888cee3294bf63f1680bdd72e427f8a696a09a\": container with ID starting with dae78cc7c8581d5fb3742253f6888cee3294bf63f1680bdd72e427f8a696a09a not found: ID does not exist" Sep 18 06:48:43.804635 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:48:43.804615 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9"] Sep 18 06:48:43.808152 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:48:43.808129 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-2f5ffe-openshift-default-5c68bc56b4-65gp9"] Sep 18 06:48:44.446668 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:48:44.446632 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc649003-f227-419f-8036-612548cba528" path="/var/lib/kubelet/pods/dc649003-f227-419f-8036-612548cba528/volumes" Sep 18 06:49:16.760773 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:49:16.760742 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8fdd5b-76c7c7b74-z97ww"] Sep 18 06:49:16.761167 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:49:16.761159 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2263b64-3f5d-4729-91e2-0975fa000742" containerName="revoke-keys" Sep 18 06:49:16.761209 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:49:16.761169 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2263b64-3f5d-4729-91e2-0975fa000742" containerName="revoke-keys" Sep 18 06:49:16.761209 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:49:16.761182 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc649003-f227-419f-8036-612548cba528" containerName="istio-proxy" Sep 18 06:49:16.761209 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:49:16.761188 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc649003-f227-419f-8036-612548cba528" containerName="istio-proxy" Sep 18 06:49:16.761209 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:49:16.761198 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="504bef64-f0e4-4856-a251-9296702689c4" containerName="istio-proxy" Sep 18 06:49:16.761209 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:49:16.761204 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="504bef64-f0e4-4856-a251-9296702689c4" containerName="istio-proxy" Sep 18 06:49:16.761382 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:49:16.761223 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1caa877a-113d-4a99-a443-c9374528c9eb" containerName="payload-pre-processing" Sep 18 06:49:16.761382 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:49:16.761228 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1caa877a-113d-4a99-a443-c9374528c9eb" containerName="payload-pre-processing" Sep 18 06:49:16.765572 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:49:16.761942 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2263b64-3f5d-4729-91e2-0975fa000742" containerName="revoke-keys" Sep 18 06:49:16.765572 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:49:16.761987 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="504bef64-f0e4-4856-a251-9296702689c4" containerName="istio-proxy" Sep 18 06:49:16.765572 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:49:16.762008 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="504bef64-f0e4-4856-a251-9296702689c4" containerName="istio-proxy" Sep 18 06:49:16.765572 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:49:16.762026 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc649003-f227-419f-8036-612548cba528" containerName="istio-proxy" Sep 18 06:49:16.765572 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:49:16.762037 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1caa877a-113d-4a99-a443-c9374528c9eb" containerName="payload-pre-processing" Sep 18 06:49:16.767968 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:49:16.767942 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8fdd5b-76c7c7b74-z97ww" Sep 18 06:49:16.772342 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:49:16.772321 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-8fdd5b\"" Sep 18 06:49:16.772668 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:49:16.772359 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-8fdd5b-dockercfg-h2lr4\"" Sep 18 06:49:16.775004 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:49:16.774984 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8fdd5b-76c7c7b74-z97ww"] Sep 18 06:49:16.793693 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:49:16.793668 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-8fdd5b-f766cfdc5-j9jrv"] Sep 18 06:49:16.794102 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:49:16.794090 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="504bef64-f0e4-4856-a251-9296702689c4" containerName="istio-proxy" Sep 18 06:49:16.794148 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:49:16.794104 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="504bef64-f0e4-4856-a251-9296702689c4" containerName="istio-proxy" Sep 18 06:49:16.797238 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:49:16.797221 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8fdd5b-f766cfdc5-j9jrv" Sep 18 06:49:16.804306 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:49:16.804277 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-8fdd5b-f766cfdc5-j9jrv"] Sep 18 06:49:16.901955 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:49:16.901918 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc417726-1f8b-4510-82e7-882bf6c02d37-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-8fdd5b-76c7c7b74-z97ww\" (UID: \"fc417726-1f8b-4510-82e7-882bf6c02d37\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8fdd5b-76c7c7b74-z97ww" Sep 18 06:49:16.902117 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:49:16.901991 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7041e5b-1691-422c-8f05-e12161264694-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-8fdd5b-f766cfdc5-j9jrv\" (UID: \"e7041e5b-1691-422c-8f05-e12161264694\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8fdd5b-f766cfdc5-j9jrv" Sep 18 06:49:16.902117 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:49:16.902018 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6trz\" (UniqueName: \"kubernetes.io/projected/e7041e5b-1691-422c-8f05-e12161264694-kube-api-access-m6trz\") pod \"payload-processing-e2e-shared-b-w3-8fdd5b-f766cfdc5-j9jrv\" (UID: \"e7041e5b-1691-422c-8f05-e12161264694\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8fdd5b-f766cfdc5-j9jrv" Sep 18 06:49:16.902117 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:49:16.902042 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59x4n\" (UniqueName: \"kubernetes.io/projected/fc417726-1f8b-4510-82e7-882bf6c02d37-kube-api-access-59x4n\") pod \"payload-pre-processing-e2e-shared-b-w3-8fdd5b-76c7c7b74-z97ww\" (UID: \"fc417726-1f8b-4510-82e7-882bf6c02d37\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8fdd5b-76c7c7b74-z97ww" Sep 18 06:49:17.003200 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:49:17.003170 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc417726-1f8b-4510-82e7-882bf6c02d37-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-8fdd5b-76c7c7b74-z97ww\" (UID: \"fc417726-1f8b-4510-82e7-882bf6c02d37\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8fdd5b-76c7c7b74-z97ww" Sep 18 06:49:17.003415 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:49:17.003259 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7041e5b-1691-422c-8f05-e12161264694-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-8fdd5b-f766cfdc5-j9jrv\" (UID: \"e7041e5b-1691-422c-8f05-e12161264694\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8fdd5b-f766cfdc5-j9jrv" Sep 18 06:49:17.003415 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:49:17.003297 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6trz\" (UniqueName: \"kubernetes.io/projected/e7041e5b-1691-422c-8f05-e12161264694-kube-api-access-m6trz\") pod \"payload-processing-e2e-shared-b-w3-8fdd5b-f766cfdc5-j9jrv\" (UID: \"e7041e5b-1691-422c-8f05-e12161264694\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8fdd5b-f766cfdc5-j9jrv" Sep 18 06:49:17.003415 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:49:17.003330 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59x4n\" (UniqueName: \"kubernetes.io/projected/fc417726-1f8b-4510-82e7-882bf6c02d37-kube-api-access-59x4n\") pod \"payload-pre-processing-e2e-shared-b-w3-8fdd5b-76c7c7b74-z97ww\" (UID: \"fc417726-1f8b-4510-82e7-882bf6c02d37\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8fdd5b-76c7c7b74-z97ww" Sep 18 06:49:17.003985 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:49:17.003966 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7041e5b-1691-422c-8f05-e12161264694-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-8fdd5b-f766cfdc5-j9jrv\" (UID: \"e7041e5b-1691-422c-8f05-e12161264694\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8fdd5b-f766cfdc5-j9jrv" Sep 18 06:49:17.003985 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:49:17.003979 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc417726-1f8b-4510-82e7-882bf6c02d37-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-8fdd5b-76c7c7b74-z97ww\" (UID: \"fc417726-1f8b-4510-82e7-882bf6c02d37\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8fdd5b-76c7c7b74-z97ww" Sep 18 06:49:17.013506 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:49:17.013439 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59x4n\" (UniqueName: \"kubernetes.io/projected/fc417726-1f8b-4510-82e7-882bf6c02d37-kube-api-access-59x4n\") pod \"payload-pre-processing-e2e-shared-b-w3-8fdd5b-76c7c7b74-z97ww\" (UID: \"fc417726-1f8b-4510-82e7-882bf6c02d37\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8fdd5b-76c7c7b74-z97ww" Sep 18 06:49:17.014052 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:49:17.014016 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6trz\" (UniqueName: \"kubernetes.io/projected/e7041e5b-1691-422c-8f05-e12161264694-kube-api-access-m6trz\") pod \"payload-processing-e2e-shared-b-w3-8fdd5b-f766cfdc5-j9jrv\" (UID: \"e7041e5b-1691-422c-8f05-e12161264694\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8fdd5b-f766cfdc5-j9jrv" Sep 18 06:49:17.082086 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:49:17.082059 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8fdd5b-76c7c7b74-z97ww" Sep 18 06:49:17.108606 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:49:17.108575 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8fdd5b-f766cfdc5-j9jrv" Sep 18 06:49:17.360430 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:49:17.360403 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8fdd5b-76c7c7b74-z97ww"] Sep 18 06:49:17.365877 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:49:17.365831 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc417726_1f8b_4510_82e7_882bf6c02d37.slice/crio-8077d91d3f3bc77ecea0956b6ee57bb3c4d496ef6815badf2a15053aa6a48620 WatchSource:0}: Error finding container 8077d91d3f3bc77ecea0956b6ee57bb3c4d496ef6815badf2a15053aa6a48620: Status 404 returned error can't find the container with id 8077d91d3f3bc77ecea0956b6ee57bb3c4d496ef6815badf2a15053aa6a48620 Sep 18 06:49:17.392632 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:49:17.392608 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-8fdd5b-f766cfdc5-j9jrv"] Sep 18 06:49:17.401439 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:49:17.401415 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7041e5b_1691_422c_8f05_e12161264694.slice/crio-6ccae1b97c08dfe115dbec06eb8fcb03f8336b69b2b62ecfc3931f4168261ca0 WatchSource:0}: Error finding container 6ccae1b97c08dfe115dbec06eb8fcb03f8336b69b2b62ecfc3931f4168261ca0: Status 404 returned error can't find the container with id 6ccae1b97c08dfe115dbec06eb8fcb03f8336b69b2b62ecfc3931f4168261ca0 Sep 18 06:49:18.002355 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:49:18.002312 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8fdd5b-76c7c7b74-z97ww" event={"ID":"fc417726-1f8b-4510-82e7-882bf6c02d37","Type":"ContainerStarted","Data":"4e214d3ab42bf319dd1508c1391ad2876e2c12bca6f2cacdfdfec5198f8eab23"} Sep 18 06:49:18.002801 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:49:18.002399 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8fdd5b-76c7c7b74-z97ww" Sep 18 06:49:18.002801 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:49:18.002425 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8fdd5b-76c7c7b74-z97ww" event={"ID":"fc417726-1f8b-4510-82e7-882bf6c02d37","Type":"ContainerStarted","Data":"8077d91d3f3bc77ecea0956b6ee57bb3c4d496ef6815badf2a15053aa6a48620"} Sep 18 06:49:18.004128 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:49:18.004107 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8fdd5b-f766cfdc5-j9jrv" event={"ID":"e7041e5b-1691-422c-8f05-e12161264694","Type":"ContainerStarted","Data":"431eb499dc6566285cb4b9e6e9f6933130e91cee78515d6e21ec27201b2ec2be"} Sep 18 06:49:18.004245 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:49:18.004144 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8fdd5b-f766cfdc5-j9jrv" event={"ID":"e7041e5b-1691-422c-8f05-e12161264694","Type":"ContainerStarted","Data":"6ccae1b97c08dfe115dbec06eb8fcb03f8336b69b2b62ecfc3931f4168261ca0"} Sep 18 06:49:18.004245 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:49:18.004215 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8fdd5b-f766cfdc5-j9jrv" Sep 18 06:49:18.019940 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:49:18.019893 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8fdd5b-76c7c7b74-z97ww" podStartSLOduration=2.019877919 podStartE2EDuration="2.019877919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:49:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:49:18.017490954 +0000 UTC m=+1619.756907156" watchObservedRunningTime="2026-09-18 06:49:18.019877919 +0000 UTC m=+1619.759294125" Sep 18 06:49:18.038763 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:49:18.038717 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8fdd5b-f766cfdc5-j9jrv" podStartSLOduration=2.038698709 podStartE2EDuration="2.038698709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:49:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:49:18.035231517 +0000 UTC m=+1619.774647715" watchObservedRunningTime="2026-09-18 06:49:18.038698709 +0000 UTC m=+1619.778114887" Sep 18 06:49:20.621813 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:49:20.621783 2567 scope.go:117] "RemoveContainer" containerID="bcb7645953980389599742ce343c881b5fa5270a3f08651004fa2084f6e9d111" Sep 18 06:49:20.631293 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:49:20.631275 2567 scope.go:117] "RemoveContainer" containerID="1777d7c4b9e38d06ee2d378bfeabe1b3fd2413d4e56126161620d6eef52d5db7" Sep 18 06:49:29.013361 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:49:29.013334 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8fdd5b-f766cfdc5-j9jrv" Sep 18 06:49:29.013796 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:49:29.013422 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8fdd5b-76c7c7b74-z97ww" Sep 18 06:50:19.879199 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:50:19.879164 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-93683709-84n9m"] Sep 18 06:50:19.882913 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:50:19.882897 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-93683709-84n9m" Sep 18 06:50:19.885872 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:50:19.885850 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 18 06:50:19.886757 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:50:19.886736 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-88t7h\"" Sep 18 06:50:19.890858 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:50:19.890834 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-93683709-84n9m"] Sep 18 06:50:19.972252 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:50:19.972224 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e6cdcdc-3606-4795-94db-0b6337434fde-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-93683709-84n9m\" (UID: \"8e6cdcdc-3606-4795-94db-0b6337434fde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-93683709-84n9m" Sep 18 06:50:20.072924 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:50:20.072900 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e6cdcdc-3606-4795-94db-0b6337434fde-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-93683709-84n9m\" (UID: \"8e6cdcdc-3606-4795-94db-0b6337434fde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-93683709-84n9m" Sep 18 06:50:20.073545 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:50:20.073527 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e6cdcdc-3606-4795-94db-0b6337434fde-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-93683709-84n9m\" (UID: \"8e6cdcdc-3606-4795-94db-0b6337434fde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-93683709-84n9m" Sep 18 06:50:20.197029 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:50:20.196959 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-93683709-84n9m" Sep 18 06:50:20.467567 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:50:20.467539 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-93683709-84n9m"] Sep 18 06:50:20.468505 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:50:20.468482 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e6cdcdc_3606_4795_94db_0b6337434fde.slice/crio-fc0db7ff2ea541c14f68f73ce045588738b0cfa4a0db7a55d83b544b7f373056 WatchSource:0}: Error finding container fc0db7ff2ea541c14f68f73ce045588738b0cfa4a0db7a55d83b544b7f373056: Status 404 returned error can't find the container with id fc0db7ff2ea541c14f68f73ce045588738b0cfa4a0db7a55d83b544b7f373056 Sep 18 06:50:20.664804 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:50:20.664781 2567 scope.go:117] "RemoveContainer" containerID="2d3292fff86cfe2f358042a205b19b2e09d586ddfe4b2d0709783624ce569d20" Sep 18 06:50:21.418585 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:50:21.418552 2567 generic.go:356] "Generic (PLEG): container finished" podID="8e6cdcdc-3606-4795-94db-0b6337434fde" containerID="ba7650920d50ce0a02a394e6f6dc0b0c54ad5c3e4521014228cc9d279f2d2c0c" exitCode=0 Sep 18 06:50:21.418953 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:50:21.418593 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-93683709-84n9m" event={"ID":"8e6cdcdc-3606-4795-94db-0b6337434fde","Type":"ContainerDied","Data":"ba7650920d50ce0a02a394e6f6dc0b0c54ad5c3e4521014228cc9d279f2d2c0c"} Sep 18 06:50:21.418953 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:50:21.418638 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-93683709-84n9m" event={"ID":"8e6cdcdc-3606-4795-94db-0b6337434fde","Type":"ContainerStarted","Data":"fc0db7ff2ea541c14f68f73ce045588738b0cfa4a0db7a55d83b544b7f373056"} Sep 18 06:50:22.563216 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:50:22.563194 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-93683709-84n9m" Sep 18 06:50:22.698524 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:50:22.698448 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e6cdcdc-3606-4795-94db-0b6337434fde-maas-api-service-ca\") pod \"8e6cdcdc-3606-4795-94db-0b6337434fde\" (UID: \"8e6cdcdc-3606-4795-94db-0b6337434fde\") " Sep 18 06:50:22.698760 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:50:22.698739 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e6cdcdc-3606-4795-94db-0b6337434fde-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8e6cdcdc-3606-4795-94db-0b6337434fde" (UID: "8e6cdcdc-3606-4795-94db-0b6337434fde"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 06:50:22.799574 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:50:22.799546 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e6cdcdc-3606-4795-94db-0b6337434fde-maas-api-service-ca\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:50:23.432384 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:50:23.432336 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-93683709-84n9m" event={"ID":"8e6cdcdc-3606-4795-94db-0b6337434fde","Type":"ContainerDied","Data":"fc0db7ff2ea541c14f68f73ce045588738b0cfa4a0db7a55d83b544b7f373056"} Sep 18 06:50:23.432553 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:50:23.432392 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-93683709-84n9m" Sep 18 06:50:23.432553 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:50:23.432409 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc0db7ff2ea541c14f68f73ce045588738b0cfa4a0db7a55d83b544b7f373056" Sep 18 06:50:35.672018 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:50:35.671925 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-93683709-84n9m"] Sep 18 06:50:35.675111 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:50:35.675082 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-93683709-84n9m"] Sep 18 06:50:36.446493 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:50:36.446463 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e6cdcdc-3606-4795-94db-0b6337434fde" path="/var/lib/kubelet/pods/8e6cdcdc-3606-4795-94db-0b6337434fde/volumes" Sep 18 06:51:20.686567 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:51:20.686537 2567 scope.go:117] "RemoveContainer" containerID="48994365af992addd259b184a572e9333033b45b9c17e49f5694b8388bab224c" Sep 18 06:51:20.698908 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:51:20.698885 2567 scope.go:117] "RemoveContainer" containerID="a30c8454b663410e7d2c73da02097705596769b015bb51bfda72aa9441b7e766" Sep 18 06:51:40.852144 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:51:40.852113 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-93683709-openshift-default-6df68648-ckchx"] Sep 18 06:51:40.852628 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:51:40.852338 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-93683709-openshift-default-6df68648-ckchx" podUID="2bd7655f-4e2a-4d12-a54f-8d26ef8eb373" containerName="istio-proxy" containerID="cri-o://95c1557c709d03768a9a38d14d80a8ce5b50a292aacd161cf57ff4ee3e5a35ad" gracePeriod=30 Sep 18 06:51:46.113639 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:51:46.113607 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-93683709-openshift-default-6df68648-ckchx" Sep 18 06:51:46.153991 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:51:46.153905 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bd7655f-4e2a-4d12-a54f-8d26ef8eb373-workload-certs\") pod \"2bd7655f-4e2a-4d12-a54f-8d26ef8eb373\" (UID: \"2bd7655f-4e2a-4d12-a54f-8d26ef8eb373\") " Sep 18 06:51:46.154152 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:51:46.153994 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bd7655f-4e2a-4d12-a54f-8d26ef8eb373-credential-socket\") pod \"2bd7655f-4e2a-4d12-a54f-8d26ef8eb373\" (UID: \"2bd7655f-4e2a-4d12-a54f-8d26ef8eb373\") " Sep 18 06:51:46.154152 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:51:46.154046 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bd7655f-4e2a-4d12-a54f-8d26ef8eb373-workload-socket\") pod \"2bd7655f-4e2a-4d12-a54f-8d26ef8eb373\" (UID: \"2bd7655f-4e2a-4d12-a54f-8d26ef8eb373\") " Sep 18 06:51:46.154152 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:51:46.154086 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bd7655f-4e2a-4d12-a54f-8d26ef8eb373-istio-podinfo\") pod \"2bd7655f-4e2a-4d12-a54f-8d26ef8eb373\" (UID: \"2bd7655f-4e2a-4d12-a54f-8d26ef8eb373\") " Sep 18 06:51:46.154152 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:51:46.154133 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bd7655f-4e2a-4d12-a54f-8d26ef8eb373-istio-token\") pod \"2bd7655f-4e2a-4d12-a54f-8d26ef8eb373\" (UID: \"2bd7655f-4e2a-4d12-a54f-8d26ef8eb373\") " Sep 18 06:51:46.154380 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:51:46.154148 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bd7655f-4e2a-4d12-a54f-8d26ef8eb373-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2bd7655f-4e2a-4d12-a54f-8d26ef8eb373" (UID: "2bd7655f-4e2a-4d12-a54f-8d26ef8eb373"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:51:46.154380 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:51:46.154260 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xttlz\" (UniqueName: \"kubernetes.io/projected/2bd7655f-4e2a-4d12-a54f-8d26ef8eb373-kube-api-access-xttlz\") pod \"2bd7655f-4e2a-4d12-a54f-8d26ef8eb373\" (UID: \"2bd7655f-4e2a-4d12-a54f-8d26ef8eb373\") " Sep 18 06:51:46.154380 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:51:46.154274 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bd7655f-4e2a-4d12-a54f-8d26ef8eb373-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2bd7655f-4e2a-4d12-a54f-8d26ef8eb373" (UID: "2bd7655f-4e2a-4d12-a54f-8d26ef8eb373"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:51:46.154380 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:51:46.154320 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bd7655f-4e2a-4d12-a54f-8d26ef8eb373-istio-envoy\") pod \"2bd7655f-4e2a-4d12-a54f-8d26ef8eb373\" (UID: \"2bd7655f-4e2a-4d12-a54f-8d26ef8eb373\") " Sep 18 06:51:46.154380 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:51:46.154376 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bd7655f-4e2a-4d12-a54f-8d26ef8eb373-istio-data\") pod \"2bd7655f-4e2a-4d12-a54f-8d26ef8eb373\" (UID: \"2bd7655f-4e2a-4d12-a54f-8d26ef8eb373\") " Sep 18 06:51:46.154593 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:51:46.154401 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bd7655f-4e2a-4d12-a54f-8d26ef8eb373-istiod-ca-cert\") pod \"2bd7655f-4e2a-4d12-a54f-8d26ef8eb373\" (UID: \"2bd7655f-4e2a-4d12-a54f-8d26ef8eb373\") " Sep 18 06:51:46.154787 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:51:46.154768 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bd7655f-4e2a-4d12-a54f-8d26ef8eb373-workload-certs\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:51:46.154827 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:51:46.154795 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bd7655f-4e2a-4d12-a54f-8d26ef8eb373-credential-socket\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:51:46.154965 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:51:46.154939 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bd7655f-4e2a-4d12-a54f-8d26ef8eb373-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2bd7655f-4e2a-4d12-a54f-8d26ef8eb373" (UID: "2bd7655f-4e2a-4d12-a54f-8d26ef8eb373"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:51:46.155148 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:51:46.155090 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bd7655f-4e2a-4d12-a54f-8d26ef8eb373-istio-data" (OuterVolumeSpecName: "istio-data") pod "2bd7655f-4e2a-4d12-a54f-8d26ef8eb373" (UID: "2bd7655f-4e2a-4d12-a54f-8d26ef8eb373"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:51:46.155261 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:51:46.155171 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bd7655f-4e2a-4d12-a54f-8d26ef8eb373-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2bd7655f-4e2a-4d12-a54f-8d26ef8eb373" (UID: "2bd7655f-4e2a-4d12-a54f-8d26ef8eb373"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 06:51:46.156914 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:51:46.156884 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bd7655f-4e2a-4d12-a54f-8d26ef8eb373-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2bd7655f-4e2a-4d12-a54f-8d26ef8eb373" (UID: "2bd7655f-4e2a-4d12-a54f-8d26ef8eb373"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:51:46.157301 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:51:46.157281 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2bd7655f-4e2a-4d12-a54f-8d26ef8eb373-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2bd7655f-4e2a-4d12-a54f-8d26ef8eb373" (UID: "2bd7655f-4e2a-4d12-a54f-8d26ef8eb373"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 06:51:46.157449 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:51:46.157320 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bd7655f-4e2a-4d12-a54f-8d26ef8eb373-istio-token" (OuterVolumeSpecName: "istio-token") pod "2bd7655f-4e2a-4d12-a54f-8d26ef8eb373" (UID: "2bd7655f-4e2a-4d12-a54f-8d26ef8eb373"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 06:51:46.157449 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:51:46.157337 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bd7655f-4e2a-4d12-a54f-8d26ef8eb373-kube-api-access-xttlz" (OuterVolumeSpecName: "kube-api-access-xttlz") pod "2bd7655f-4e2a-4d12-a54f-8d26ef8eb373" (UID: "2bd7655f-4e2a-4d12-a54f-8d26ef8eb373"). InnerVolumeSpecName "kube-api-access-xttlz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 06:51:46.255739 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:51:46.255711 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bd7655f-4e2a-4d12-a54f-8d26ef8eb373-workload-socket\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:51:46.255739 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:51:46.255736 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bd7655f-4e2a-4d12-a54f-8d26ef8eb373-istio-podinfo\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:51:46.255909 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:51:46.255745 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bd7655f-4e2a-4d12-a54f-8d26ef8eb373-istio-token\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:51:46.255909 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:51:46.255755 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xttlz\" (UniqueName: \"kubernetes.io/projected/2bd7655f-4e2a-4d12-a54f-8d26ef8eb373-kube-api-access-xttlz\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:51:46.255909 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:51:46.255764 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bd7655f-4e2a-4d12-a54f-8d26ef8eb373-istio-envoy\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:51:46.255909 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:51:46.255772 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bd7655f-4e2a-4d12-a54f-8d26ef8eb373-istio-data\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:51:46.255909 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:51:46.255781 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bd7655f-4e2a-4d12-a54f-8d26ef8eb373-istiod-ca-cert\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:51:46.969361 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:51:46.969321 2567 generic.go:356] "Generic (PLEG): container finished" podID="2bd7655f-4e2a-4d12-a54f-8d26ef8eb373" containerID="95c1557c709d03768a9a38d14d80a8ce5b50a292aacd161cf57ff4ee3e5a35ad" exitCode=0 Sep 18 06:51:46.969553 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:51:46.969396 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-93683709-openshift-default-6df68648-ckchx" event={"ID":"2bd7655f-4e2a-4d12-a54f-8d26ef8eb373","Type":"ContainerDied","Data":"95c1557c709d03768a9a38d14d80a8ce5b50a292aacd161cf57ff4ee3e5a35ad"} Sep 18 06:51:46.969553 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:51:46.969413 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-93683709-openshift-default-6df68648-ckchx" Sep 18 06:51:46.969553 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:51:46.969483 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-93683709-openshift-default-6df68648-ckchx" event={"ID":"2bd7655f-4e2a-4d12-a54f-8d26ef8eb373","Type":"ContainerDied","Data":"a35f195ebd4828dafb806e34b8173692bb75ed2fb28ca7c24cc831900e13340e"} Sep 18 06:51:46.969553 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:51:46.969507 2567 scope.go:117] "RemoveContainer" containerID="95c1557c709d03768a9a38d14d80a8ce5b50a292aacd161cf57ff4ee3e5a35ad" Sep 18 06:51:46.979660 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:51:46.979453 2567 scope.go:117] "RemoveContainer" containerID="dd8532918004f2e04ec69a11fcc99ccf99c22a2f7b08ce4e2bfc5a85b1eb91b8" Sep 18 06:51:46.989668 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:51:46.989641 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-93683709-openshift-default-6df68648-ckchx"] Sep 18 06:51:46.991270 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:51:46.991252 2567 scope.go:117] "RemoveContainer" containerID="95c1557c709d03768a9a38d14d80a8ce5b50a292aacd161cf57ff4ee3e5a35ad" Sep 18 06:51:46.991559 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:51:46.991533 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95c1557c709d03768a9a38d14d80a8ce5b50a292aacd161cf57ff4ee3e5a35ad\": container with ID starting with 95c1557c709d03768a9a38d14d80a8ce5b50a292aacd161cf57ff4ee3e5a35ad not found: ID does not exist" containerID="95c1557c709d03768a9a38d14d80a8ce5b50a292aacd161cf57ff4ee3e5a35ad" Sep 18 06:51:46.991657 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:51:46.991571 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95c1557c709d03768a9a38d14d80a8ce5b50a292aacd161cf57ff4ee3e5a35ad"} err="failed to get container status \"95c1557c709d03768a9a38d14d80a8ce5b50a292aacd161cf57ff4ee3e5a35ad\": rpc error: code = NotFound desc = could not find container \"95c1557c709d03768a9a38d14d80a8ce5b50a292aacd161cf57ff4ee3e5a35ad\": container with ID starting with 95c1557c709d03768a9a38d14d80a8ce5b50a292aacd161cf57ff4ee3e5a35ad not found: ID does not exist" Sep 18 06:51:46.991657 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:51:46.991595 2567 scope.go:117] "RemoveContainer" containerID="dd8532918004f2e04ec69a11fcc99ccf99c22a2f7b08ce4e2bfc5a85b1eb91b8" Sep 18 06:51:46.991867 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:51:46.991848 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd8532918004f2e04ec69a11fcc99ccf99c22a2f7b08ce4e2bfc5a85b1eb91b8\": container with ID starting with dd8532918004f2e04ec69a11fcc99ccf99c22a2f7b08ce4e2bfc5a85b1eb91b8 not found: ID does not exist" containerID="dd8532918004f2e04ec69a11fcc99ccf99c22a2f7b08ce4e2bfc5a85b1eb91b8" Sep 18 06:51:46.991908 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:51:46.991873 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd8532918004f2e04ec69a11fcc99ccf99c22a2f7b08ce4e2bfc5a85b1eb91b8"} err="failed to get container status \"dd8532918004f2e04ec69a11fcc99ccf99c22a2f7b08ce4e2bfc5a85b1eb91b8\": rpc error: code = NotFound desc = could not find container \"dd8532918004f2e04ec69a11fcc99ccf99c22a2f7b08ce4e2bfc5a85b1eb91b8\": container with ID starting with dd8532918004f2e04ec69a11fcc99ccf99c22a2f7b08ce4e2bfc5a85b1eb91b8 not found: ID does not exist" Sep 18 06:51:46.993189 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:51:46.993171 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-93683709-openshift-default-6df68648-ckchx"] Sep 18 06:51:48.446513 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:51:48.446476 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bd7655f-4e2a-4d12-a54f-8d26ef8eb373" path="/var/lib/kubelet/pods/2bd7655f-4e2a-4d12-a54f-8d26ef8eb373/volumes" Sep 18 06:52:20.551820 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:20.551791 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjr4h_eb059951-e263-40da-9155-4fd0cf242820/ovn-acl-logging/0.log" Sep 18 06:52:20.558419 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:20.558398 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjr4h_eb059951-e263-40da-9155-4fd0cf242820/ovn-acl-logging/0.log" Sep 18 06:52:20.733696 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:20.733669 2567 scope.go:117] "RemoveContainer" containerID="96463803e08debcf3a780e8caa47a5f7d4380455812521a46e9dc6a0079bc6d8" Sep 18 06:52:34.456281 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:34.456243 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9"] Sep 18 06:52:34.457016 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:34.456955 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e6cdcdc-3606-4795-94db-0b6337434fde" containerName="revoke-keys" Sep 18 06:52:34.457016 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:34.456974 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e6cdcdc-3606-4795-94db-0b6337434fde" containerName="revoke-keys" Sep 18 06:52:34.457016 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:34.456994 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bd7655f-4e2a-4d12-a54f-8d26ef8eb373" containerName="istio-proxy" Sep 18 06:52:34.457016 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:34.457003 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bd7655f-4e2a-4d12-a54f-8d26ef8eb373" containerName="istio-proxy" Sep 18 06:52:34.457282 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:34.457023 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bd7655f-4e2a-4d12-a54f-8d26ef8eb373" containerName="istio-proxy" Sep 18 06:52:34.457282 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:34.457032 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bd7655f-4e2a-4d12-a54f-8d26ef8eb373" containerName="istio-proxy" Sep 18 06:52:34.457282 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:34.457116 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bd7655f-4e2a-4d12-a54f-8d26ef8eb373" containerName="istio-proxy" Sep 18 06:52:34.457282 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:34.457127 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e6cdcdc-3606-4795-94db-0b6337434fde" containerName="revoke-keys" Sep 18 06:52:34.462416 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:34.462399 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9" Sep 18 06:52:34.464761 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:34.464742 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-2686de-openshift-default-dockercfg-skxn4\"" Sep 18 06:52:34.473923 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:34.473902 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9"] Sep 18 06:52:34.582190 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:34.582143 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/324189bf-a066-41d2-a81b-ae9df0df7114-istio-token\") pod \"e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9\" (UID: \"324189bf-a066-41d2-a81b-ae9df0df7114\") " pod="openshift-ingress/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9" Sep 18 06:52:34.582525 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:34.582497 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/324189bf-a066-41d2-a81b-ae9df0df7114-credential-socket\") pod \"e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9\" (UID: \"324189bf-a066-41d2-a81b-ae9df0df7114\") " pod="openshift-ingress/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9" Sep 18 06:52:34.582667 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:34.582621 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/324189bf-a066-41d2-a81b-ae9df0df7114-workload-certs\") pod \"e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9\" (UID: \"324189bf-a066-41d2-a81b-ae9df0df7114\") " pod="openshift-ingress/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9" Sep 18 06:52:34.582737 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:34.582664 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/324189bf-a066-41d2-a81b-ae9df0df7114-istio-podinfo\") pod \"e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9\" (UID: \"324189bf-a066-41d2-a81b-ae9df0df7114\") " pod="openshift-ingress/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9" Sep 18 06:52:34.582737 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:34.582689 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/324189bf-a066-41d2-a81b-ae9df0df7114-istiod-ca-cert\") pod \"e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9\" (UID: \"324189bf-a066-41d2-a81b-ae9df0df7114\") " pod="openshift-ingress/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9" Sep 18 06:52:34.582737 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:34.582714 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx4t4\" (UniqueName: \"kubernetes.io/projected/324189bf-a066-41d2-a81b-ae9df0df7114-kube-api-access-sx4t4\") pod \"e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9\" (UID: \"324189bf-a066-41d2-a81b-ae9df0df7114\") " pod="openshift-ingress/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9" Sep 18 06:52:34.582895 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:34.582743 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/324189bf-a066-41d2-a81b-ae9df0df7114-istio-data\") pod \"e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9\" (UID: \"324189bf-a066-41d2-a81b-ae9df0df7114\") " pod="openshift-ingress/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9" Sep 18 06:52:34.582895 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:34.582868 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/324189bf-a066-41d2-a81b-ae9df0df7114-workload-socket\") pod \"e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9\" (UID: \"324189bf-a066-41d2-a81b-ae9df0df7114\") " pod="openshift-ingress/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9" Sep 18 06:52:34.583003 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:34.582920 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/324189bf-a066-41d2-a81b-ae9df0df7114-istio-envoy\") pod \"e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9\" (UID: \"324189bf-a066-41d2-a81b-ae9df0df7114\") " pod="openshift-ingress/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9" Sep 18 06:52:34.684087 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:34.684049 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/324189bf-a066-41d2-a81b-ae9df0df7114-workload-socket\") pod \"e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9\" (UID: \"324189bf-a066-41d2-a81b-ae9df0df7114\") " pod="openshift-ingress/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9" Sep 18 06:52:34.684279 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:34.684105 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/324189bf-a066-41d2-a81b-ae9df0df7114-istio-envoy\") pod \"e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9\" (UID: \"324189bf-a066-41d2-a81b-ae9df0df7114\") " pod="openshift-ingress/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9" Sep 18 06:52:34.684279 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:34.684151 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/324189bf-a066-41d2-a81b-ae9df0df7114-istio-token\") pod \"e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9\" (UID: \"324189bf-a066-41d2-a81b-ae9df0df7114\") " pod="openshift-ingress/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9" Sep 18 06:52:34.684279 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:34.684177 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/324189bf-a066-41d2-a81b-ae9df0df7114-credential-socket\") pod \"e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9\" (UID: \"324189bf-a066-41d2-a81b-ae9df0df7114\") " pod="openshift-ingress/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9" Sep 18 06:52:34.684279 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:34.684231 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/324189bf-a066-41d2-a81b-ae9df0df7114-workload-certs\") pod \"e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9\" (UID: \"324189bf-a066-41d2-a81b-ae9df0df7114\") " pod="openshift-ingress/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9" Sep 18 06:52:34.684279 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:34.684264 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/324189bf-a066-41d2-a81b-ae9df0df7114-istio-podinfo\") pod \"e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9\" (UID: \"324189bf-a066-41d2-a81b-ae9df0df7114\") " pod="openshift-ingress/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9" Sep 18 06:52:34.684602 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:34.684291 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/324189bf-a066-41d2-a81b-ae9df0df7114-istiod-ca-cert\") pod \"e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9\" (UID: \"324189bf-a066-41d2-a81b-ae9df0df7114\") " pod="openshift-ingress/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9" Sep 18 06:52:34.684602 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:34.684317 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx4t4\" (UniqueName: \"kubernetes.io/projected/324189bf-a066-41d2-a81b-ae9df0df7114-kube-api-access-sx4t4\") pod \"e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9\" (UID: \"324189bf-a066-41d2-a81b-ae9df0df7114\") " pod="openshift-ingress/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9" Sep 18 06:52:34.684602 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:34.684346 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/324189bf-a066-41d2-a81b-ae9df0df7114-istio-data\") pod \"e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9\" (UID: \"324189bf-a066-41d2-a81b-ae9df0df7114\") " pod="openshift-ingress/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9" Sep 18 06:52:34.684602 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:34.684528 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/324189bf-a066-41d2-a81b-ae9df0df7114-workload-socket\") pod \"e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9\" (UID: \"324189bf-a066-41d2-a81b-ae9df0df7114\") " pod="openshift-ingress/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9" Sep 18 06:52:34.684840 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:34.684683 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/324189bf-a066-41d2-a81b-ae9df0df7114-workload-certs\") pod \"e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9\" (UID: \"324189bf-a066-41d2-a81b-ae9df0df7114\") " pod="openshift-ingress/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9" Sep 18 06:52:34.684840 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:34.684721 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/324189bf-a066-41d2-a81b-ae9df0df7114-credential-socket\") pod \"e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9\" (UID: \"324189bf-a066-41d2-a81b-ae9df0df7114\") " pod="openshift-ingress/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9" Sep 18 06:52:34.684840 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:34.684755 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/324189bf-a066-41d2-a81b-ae9df0df7114-istio-data\") pod \"e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9\" (UID: \"324189bf-a066-41d2-a81b-ae9df0df7114\") " pod="openshift-ingress/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9" Sep 18 06:52:34.685157 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:34.685139 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/324189bf-a066-41d2-a81b-ae9df0df7114-istiod-ca-cert\") pod \"e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9\" (UID: \"324189bf-a066-41d2-a81b-ae9df0df7114\") " pod="openshift-ingress/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9" Sep 18 06:52:34.686468 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:34.686453 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/324189bf-a066-41d2-a81b-ae9df0df7114-istio-envoy\") pod \"e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9\" (UID: \"324189bf-a066-41d2-a81b-ae9df0df7114\") " pod="openshift-ingress/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9" Sep 18 06:52:34.686783 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:34.686765 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/324189bf-a066-41d2-a81b-ae9df0df7114-istio-podinfo\") pod \"e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9\" (UID: \"324189bf-a066-41d2-a81b-ae9df0df7114\") " pod="openshift-ingress/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9" Sep 18 06:52:34.692076 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:34.692049 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/324189bf-a066-41d2-a81b-ae9df0df7114-istio-token\") pod \"e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9\" (UID: \"324189bf-a066-41d2-a81b-ae9df0df7114\") " pod="openshift-ingress/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9" Sep 18 06:52:34.692232 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:34.692211 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx4t4\" (UniqueName: \"kubernetes.io/projected/324189bf-a066-41d2-a81b-ae9df0df7114-kube-api-access-sx4t4\") pod \"e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9\" (UID: \"324189bf-a066-41d2-a81b-ae9df0df7114\") " pod="openshift-ingress/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9" Sep 18 06:52:34.774542 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:34.774488 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9" Sep 18 06:52:35.060000 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:35.059963 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9"] Sep 18 06:52:35.060922 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:52:35.060891 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod324189bf_a066_41d2_a81b_ae9df0df7114.slice/crio-3d788ef8f4ea56d50d2574aa523b819fb29003b9002b1134d9499a7335541006 WatchSource:0}: Error finding container 3d788ef8f4ea56d50d2574aa523b819fb29003b9002b1134d9499a7335541006: Status 404 returned error can't find the container with id 3d788ef8f4ea56d50d2574aa523b819fb29003b9002b1134d9499a7335541006 Sep 18 06:52:35.063295 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:35.063263 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:52:35.063414 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:35.063327 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:52:35.063414 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:35.063359 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:52:35.268728 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:35.268691 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9" event={"ID":"324189bf-a066-41d2-a81b-ae9df0df7114","Type":"ContainerStarted","Data":"cd5f3cfe43edb8de043a074c1338febb1af692c93fefb46831b3abcfb260dc48"} Sep 18 06:52:35.268928 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:35.268752 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9" event={"ID":"324189bf-a066-41d2-a81b-ae9df0df7114","Type":"ContainerStarted","Data":"3d788ef8f4ea56d50d2574aa523b819fb29003b9002b1134d9499a7335541006"} Sep 18 06:52:35.287953 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:35.287904 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9" podStartSLOduration=1.287890078 podStartE2EDuration="1.287890078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:52:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:52:35.285596669 +0000 UTC m=+1817.025012868" watchObservedRunningTime="2026-09-18 06:52:35.287890078 +0000 UTC m=+1817.027306253" Sep 18 06:52:35.774880 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:35.774847 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9" Sep 18 06:52:36.780588 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:36.780555 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9" Sep 18 06:52:37.284164 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:37.284127 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9" Sep 18 06:52:37.285468 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:37.285439 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9" Sep 18 06:52:42.616668 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:42.616639 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2686de-549bdbcdb9-bzgvc"] Sep 18 06:52:42.617174 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:42.617161 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bd7655f-4e2a-4d12-a54f-8d26ef8eb373" containerName="istio-proxy" Sep 18 06:52:42.620400 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:42.620383 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2686de-549bdbcdb9-bzgvc" Sep 18 06:52:42.622928 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:42.622906 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-2686de-dockercfg-6dppf\"" Sep 18 06:52:42.623052 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:42.622902 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-2686de\"" Sep 18 06:52:42.628338 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:42.628309 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2686de-549bdbcdb9-bzgvc"] Sep 18 06:52:42.775918 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:42.775885 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cc8m\" (UniqueName: \"kubernetes.io/projected/5b36a5fd-29ba-48fe-9884-a31a22691fd1-kube-api-access-8cc8m\") pod \"payload-processing-e2e-api-a-2686de-549bdbcdb9-bzgvc\" (UID: \"5b36a5fd-29ba-48fe-9884-a31a22691fd1\") " pod="openshift-ingress/payload-processing-e2e-api-a-2686de-549bdbcdb9-bzgvc" Sep 18 06:52:42.776075 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:42.775930 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b36a5fd-29ba-48fe-9884-a31a22691fd1-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2686de-549bdbcdb9-bzgvc\" (UID: \"5b36a5fd-29ba-48fe-9884-a31a22691fd1\") " pod="openshift-ingress/payload-processing-e2e-api-a-2686de-549bdbcdb9-bzgvc" Sep 18 06:52:42.877215 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:42.877132 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cc8m\" (UniqueName: \"kubernetes.io/projected/5b36a5fd-29ba-48fe-9884-a31a22691fd1-kube-api-access-8cc8m\") pod \"payload-processing-e2e-api-a-2686de-549bdbcdb9-bzgvc\" (UID: \"5b36a5fd-29ba-48fe-9884-a31a22691fd1\") " pod="openshift-ingress/payload-processing-e2e-api-a-2686de-549bdbcdb9-bzgvc" Sep 18 06:52:42.877215 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:42.877184 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b36a5fd-29ba-48fe-9884-a31a22691fd1-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2686de-549bdbcdb9-bzgvc\" (UID: \"5b36a5fd-29ba-48fe-9884-a31a22691fd1\") " pod="openshift-ingress/payload-processing-e2e-api-a-2686de-549bdbcdb9-bzgvc" Sep 18 06:52:42.877946 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:42.877928 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b36a5fd-29ba-48fe-9884-a31a22691fd1-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2686de-549bdbcdb9-bzgvc\" (UID: \"5b36a5fd-29ba-48fe-9884-a31a22691fd1\") " pod="openshift-ingress/payload-processing-e2e-api-a-2686de-549bdbcdb9-bzgvc" Sep 18 06:52:42.884977 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:42.884951 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cc8m\" (UniqueName: \"kubernetes.io/projected/5b36a5fd-29ba-48fe-9884-a31a22691fd1-kube-api-access-8cc8m\") pod \"payload-processing-e2e-api-a-2686de-549bdbcdb9-bzgvc\" (UID: \"5b36a5fd-29ba-48fe-9884-a31a22691fd1\") " pod="openshift-ingress/payload-processing-e2e-api-a-2686de-549bdbcdb9-bzgvc" Sep 18 06:52:42.932923 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:42.932900 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2686de-549bdbcdb9-bzgvc" Sep 18 06:52:43.203741 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:43.203712 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2686de-549bdbcdb9-bzgvc"] Sep 18 06:52:43.204439 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:52:43.204413 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b36a5fd_29ba_48fe_9884_a31a22691fd1.slice/crio-82c58709bc520db4831169e0d329c8f1bddba39cceb3f3ab3ca038f0122661de WatchSource:0}: Error finding container 82c58709bc520db4831169e0d329c8f1bddba39cceb3f3ab3ca038f0122661de: Status 404 returned error can't find the container with id 82c58709bc520db4831169e0d329c8f1bddba39cceb3f3ab3ca038f0122661de Sep 18 06:52:43.325789 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:43.325754 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2686de-549bdbcdb9-bzgvc" event={"ID":"5b36a5fd-29ba-48fe-9884-a31a22691fd1","Type":"ContainerStarted","Data":"c5cb7d3f072dacf335c28c8efa45db29909cf7958f6f80f992cef031f4915422"} Sep 18 06:52:43.325958 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:43.325816 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2686de-549bdbcdb9-bzgvc" event={"ID":"5b36a5fd-29ba-48fe-9884-a31a22691fd1","Type":"ContainerStarted","Data":"82c58709bc520db4831169e0d329c8f1bddba39cceb3f3ab3ca038f0122661de"} Sep 18 06:52:43.325958 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:43.325883 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-2686de-549bdbcdb9-bzgvc" Sep 18 06:52:43.354253 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:43.354198 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-2686de-549bdbcdb9-bzgvc" podStartSLOduration=1.354172553 podStartE2EDuration="1.354172553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:52:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:52:43.353228421 +0000 UTC m=+1825.092644620" watchObservedRunningTime="2026-09-18 06:52:43.354172553 +0000 UTC m=+1825.093588770" Sep 18 06:52:54.336021 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:52:54.335987 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-2686de-549bdbcdb9-bzgvc" Sep 18 06:53:03.454570 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:03.454537 2567 generic.go:356] "Generic (PLEG): container finished" podID="324189bf-a066-41d2-a81b-ae9df0df7114" containerID="cd5f3cfe43edb8de043a074c1338febb1af692c93fefb46831b3abcfb260dc48" exitCode=0 Sep 18 06:53:03.454977 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:03.454609 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9" event={"ID":"324189bf-a066-41d2-a81b-ae9df0df7114","Type":"ContainerDied","Data":"cd5f3cfe43edb8de043a074c1338febb1af692c93fefb46831b3abcfb260dc48"} Sep 18 06:53:03.455020 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:03.454988 2567 scope.go:117] "RemoveContainer" containerID="cd5f3cfe43edb8de043a074c1338febb1af692c93fefb46831b3abcfb260dc48" Sep 18 06:53:03.456168 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:03.456135 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:53:03.456223 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:03.456210 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:53:03.456299 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:03.456240 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:53:04.473105 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:04.473067 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9" event={"ID":"324189bf-a066-41d2-a81b-ae9df0df7114","Type":"ContainerStarted","Data":"8221a6a83850acb66995c7cce4ea6ff1dddea04815a7e3645d6b5e91b81980b2"} Sep 18 06:53:04.775440 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:04.775312 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9" Sep 18 06:53:04.776645 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:04.776618 2567 patch_prober.go:28] interesting pod/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Sep 18 06:53:04.776763 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:04.776683 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9" podUID="324189bf-a066-41d2-a81b-ae9df0df7114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Sep 18 06:53:05.775247 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:05.775209 2567 patch_prober.go:28] interesting pod/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Sep 18 06:53:05.775645 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:05.775278 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9" podUID="324189bf-a066-41d2-a81b-ae9df0df7114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Sep 18 06:53:06.775204 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:06.775172 2567 patch_prober.go:28] interesting pod/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Sep 18 06:53:06.775389 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:06.775235 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9" podUID="324189bf-a066-41d2-a81b-ae9df0df7114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Sep 18 06:53:07.284790 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:07.284759 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9" Sep 18 06:53:07.775044 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:07.775009 2567 patch_prober.go:28] interesting pod/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Sep 18 06:53:07.775224 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:07.775062 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9" podUID="324189bf-a066-41d2-a81b-ae9df0df7114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Sep 18 06:53:07.881258 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:07.881208 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d"] Sep 18 06:53:07.885791 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:07.885766 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d" Sep 18 06:53:07.889317 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:07.889291 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d"] Sep 18 06:53:07.892771 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:07.892749 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-bcb767-openshift-default-dockercfg-tbvdf\"" Sep 18 06:53:07.920599 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:07.920570 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a58e0899-5ec2-4ca5-b517-f3929441a513-istio-token\") pod \"e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d\" (UID: \"a58e0899-5ec2-4ca5-b517-f3929441a513\") " pod="openshift-ingress/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d" Sep 18 06:53:07.920717 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:07.920619 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a58e0899-5ec2-4ca5-b517-f3929441a513-workload-socket\") pod \"e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d\" (UID: \"a58e0899-5ec2-4ca5-b517-f3929441a513\") " pod="openshift-ingress/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d" Sep 18 06:53:07.920717 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:07.920651 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a58e0899-5ec2-4ca5-b517-f3929441a513-credential-socket\") pod \"e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d\" (UID: \"a58e0899-5ec2-4ca5-b517-f3929441a513\") " pod="openshift-ingress/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d" Sep 18 06:53:07.920717 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:07.920672 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf285\" (UniqueName: \"kubernetes.io/projected/a58e0899-5ec2-4ca5-b517-f3929441a513-kube-api-access-vf285\") pod \"e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d\" (UID: \"a58e0899-5ec2-4ca5-b517-f3929441a513\") " pod="openshift-ingress/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d" Sep 18 06:53:07.920717 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:07.920691 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a58e0899-5ec2-4ca5-b517-f3929441a513-istio-envoy\") pod \"e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d\" (UID: \"a58e0899-5ec2-4ca5-b517-f3929441a513\") " pod="openshift-ingress/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d" Sep 18 06:53:07.920868 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:07.920719 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a58e0899-5ec2-4ca5-b517-f3929441a513-istio-podinfo\") pod \"e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d\" (UID: \"a58e0899-5ec2-4ca5-b517-f3929441a513\") " pod="openshift-ingress/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d" Sep 18 06:53:07.920868 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:07.920779 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a58e0899-5ec2-4ca5-b517-f3929441a513-istiod-ca-cert\") pod \"e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d\" (UID: \"a58e0899-5ec2-4ca5-b517-f3929441a513\") " pod="openshift-ingress/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d" Sep 18 06:53:07.920868 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:07.920841 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a58e0899-5ec2-4ca5-b517-f3929441a513-workload-certs\") pod \"e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d\" (UID: \"a58e0899-5ec2-4ca5-b517-f3929441a513\") " pod="openshift-ingress/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d" Sep 18 06:53:07.920868 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:07.920861 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a58e0899-5ec2-4ca5-b517-f3929441a513-istio-data\") pod \"e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d\" (UID: \"a58e0899-5ec2-4ca5-b517-f3929441a513\") " pod="openshift-ingress/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d" Sep 18 06:53:08.022004 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:08.021965 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a58e0899-5ec2-4ca5-b517-f3929441a513-workload-socket\") pod \"e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d\" (UID: \"a58e0899-5ec2-4ca5-b517-f3929441a513\") " pod="openshift-ingress/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d" Sep 18 06:53:08.022004 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:08.022011 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a58e0899-5ec2-4ca5-b517-f3929441a513-credential-socket\") pod \"e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d\" (UID: \"a58e0899-5ec2-4ca5-b517-f3929441a513\") " pod="openshift-ingress/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d" Sep 18 06:53:08.022260 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:08.022036 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf285\" (UniqueName: \"kubernetes.io/projected/a58e0899-5ec2-4ca5-b517-f3929441a513-kube-api-access-vf285\") pod \"e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d\" (UID: \"a58e0899-5ec2-4ca5-b517-f3929441a513\") " pod="openshift-ingress/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d" Sep 18 06:53:08.022260 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:08.022060 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a58e0899-5ec2-4ca5-b517-f3929441a513-istio-envoy\") pod \"e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d\" (UID: \"a58e0899-5ec2-4ca5-b517-f3929441a513\") " pod="openshift-ingress/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d" Sep 18 06:53:08.022260 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:08.022080 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a58e0899-5ec2-4ca5-b517-f3929441a513-istio-podinfo\") pod \"e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d\" (UID: \"a58e0899-5ec2-4ca5-b517-f3929441a513\") " pod="openshift-ingress/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d" Sep 18 06:53:08.022260 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:08.022105 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a58e0899-5ec2-4ca5-b517-f3929441a513-istiod-ca-cert\") pod \"e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d\" (UID: \"a58e0899-5ec2-4ca5-b517-f3929441a513\") " pod="openshift-ingress/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d" Sep 18 06:53:08.022260 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:08.022163 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a58e0899-5ec2-4ca5-b517-f3929441a513-workload-certs\") pod \"e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d\" (UID: \"a58e0899-5ec2-4ca5-b517-f3929441a513\") " pod="openshift-ingress/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d" Sep 18 06:53:08.022260 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:08.022188 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a58e0899-5ec2-4ca5-b517-f3929441a513-istio-data\") pod \"e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d\" (UID: \"a58e0899-5ec2-4ca5-b517-f3929441a513\") " pod="openshift-ingress/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d" Sep 18 06:53:08.022260 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:08.022245 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a58e0899-5ec2-4ca5-b517-f3929441a513-istio-token\") pod \"e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d\" (UID: \"a58e0899-5ec2-4ca5-b517-f3929441a513\") " pod="openshift-ingress/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d" Sep 18 06:53:08.022663 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:08.022392 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a58e0899-5ec2-4ca5-b517-f3929441a513-workload-socket\") pod \"e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d\" (UID: \"a58e0899-5ec2-4ca5-b517-f3929441a513\") " pod="openshift-ingress/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d" Sep 18 06:53:08.022663 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:08.022449 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a58e0899-5ec2-4ca5-b517-f3929441a513-credential-socket\") pod \"e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d\" (UID: \"a58e0899-5ec2-4ca5-b517-f3929441a513\") " pod="openshift-ingress/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d" Sep 18 06:53:08.022663 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:08.022601 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a58e0899-5ec2-4ca5-b517-f3929441a513-workload-certs\") pod \"e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d\" (UID: \"a58e0899-5ec2-4ca5-b517-f3929441a513\") " pod="openshift-ingress/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d" Sep 18 06:53:08.022838 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:08.022692 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a58e0899-5ec2-4ca5-b517-f3929441a513-istio-data\") pod \"e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d\" (UID: \"a58e0899-5ec2-4ca5-b517-f3929441a513\") " pod="openshift-ingress/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d" Sep 18 06:53:08.022885 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:08.022859 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a58e0899-5ec2-4ca5-b517-f3929441a513-istiod-ca-cert\") pod \"e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d\" (UID: \"a58e0899-5ec2-4ca5-b517-f3929441a513\") " pod="openshift-ingress/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d" Sep 18 06:53:08.024649 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:08.024627 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a58e0899-5ec2-4ca5-b517-f3929441a513-istio-envoy\") pod \"e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d\" (UID: \"a58e0899-5ec2-4ca5-b517-f3929441a513\") " pod="openshift-ingress/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d" Sep 18 06:53:08.024778 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:08.024754 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a58e0899-5ec2-4ca5-b517-f3929441a513-istio-podinfo\") pod \"e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d\" (UID: \"a58e0899-5ec2-4ca5-b517-f3929441a513\") " pod="openshift-ingress/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d" Sep 18 06:53:08.040034 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:08.039978 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a58e0899-5ec2-4ca5-b517-f3929441a513-istio-token\") pod \"e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d\" (UID: \"a58e0899-5ec2-4ca5-b517-f3929441a513\") " pod="openshift-ingress/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d" Sep 18 06:53:08.040154 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:08.040112 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf285\" (UniqueName: \"kubernetes.io/projected/a58e0899-5ec2-4ca5-b517-f3929441a513-kube-api-access-vf285\") pod \"e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d\" (UID: \"a58e0899-5ec2-4ca5-b517-f3929441a513\") " pod="openshift-ingress/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d" Sep 18 06:53:08.203230 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:08.203198 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d" Sep 18 06:53:08.506951 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:08.506919 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d"] Sep 18 06:53:08.507974 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:53:08.507950 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda58e0899_5ec2_4ca5_b517_f3929441a513.slice/crio-d6fc4fc93d6fd1406a8c762b5ac5b9025dd7de0d54c1e1d70a50b553318bc2c8 WatchSource:0}: Error finding container d6fc4fc93d6fd1406a8c762b5ac5b9025dd7de0d54c1e1d70a50b553318bc2c8: Status 404 returned error can't find the container with id d6fc4fc93d6fd1406a8c762b5ac5b9025dd7de0d54c1e1d70a50b553318bc2c8 Sep 18 06:53:08.510182 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:08.510148 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:53:08.510275 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:08.510227 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:53:08.510337 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:08.510272 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:53:08.775707 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:08.775611 2567 patch_prober.go:28] interesting pod/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Sep 18 06:53:08.776398 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:08.775977 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9" podUID="324189bf-a066-41d2-a81b-ae9df0df7114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Sep 18 06:53:09.506969 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:09.506928 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d" event={"ID":"a58e0899-5ec2-4ca5-b517-f3929441a513","Type":"ContainerStarted","Data":"af0536b8581261eb26e1911e939d9aac2ba6cdd4360fd20b9de2da96787220a3"} Sep 18 06:53:09.507430 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:09.506990 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d" event={"ID":"a58e0899-5ec2-4ca5-b517-f3929441a513","Type":"ContainerStarted","Data":"d6fc4fc93d6fd1406a8c762b5ac5b9025dd7de0d54c1e1d70a50b553318bc2c8"} Sep 18 06:53:09.528708 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:09.528654 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d" podStartSLOduration=2.528639334 podStartE2EDuration="2.528639334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:53:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:53:09.526183081 +0000 UTC m=+1851.265599279" watchObservedRunningTime="2026-09-18 06:53:09.528639334 +0000 UTC m=+1851.268055508" Sep 18 06:53:09.779501 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:09.779427 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9" Sep 18 06:53:09.780314 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:09.780294 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9" Sep 18 06:53:10.204089 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:10.204053 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d" Sep 18 06:53:10.209034 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:10.209011 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d" Sep 18 06:53:10.513355 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:10.513268 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d" Sep 18 06:53:10.514126 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:10.514105 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d" Sep 18 06:53:16.925226 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:16.925190 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bcb767-ddbf7877d-dvwlf"] Sep 18 06:53:16.928043 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:16.928027 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bcb767-ddbf7877d-dvwlf" Sep 18 06:53:16.931093 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:16.931072 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-bcb767-dockercfg-292r5\"" Sep 18 06:53:16.931093 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:16.931087 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-bcb767\"" Sep 18 06:53:16.938496 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:16.938473 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bcb767-ddbf7877d-dvwlf"] Sep 18 06:53:16.960193 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:16.960171 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bcb767-86b5b575f9-lltb5"] Sep 18 06:53:16.962748 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:16.962733 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bcb767-86b5b575f9-lltb5" Sep 18 06:53:16.972779 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:16.972755 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bcb767-86b5b575f9-lltb5"] Sep 18 06:53:17.010187 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:17.010160 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fbjq\" (UniqueName: \"kubernetes.io/projected/0518cd5e-3515-4b8b-8865-c638876b77e2-kube-api-access-8fbjq\") pod \"payload-processing-e2e-api-b-bcb767-86b5b575f9-lltb5\" (UID: \"0518cd5e-3515-4b8b-8865-c638876b77e2\") " pod="openshift-ingress/payload-processing-e2e-api-b-bcb767-86b5b575f9-lltb5" Sep 18 06:53:17.010338 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:17.010208 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48708d6d-8b5e-4a16-b3b1-22bf2208d3c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-bcb767-ddbf7877d-dvwlf\" (UID: \"48708d6d-8b5e-4a16-b3b1-22bf2208d3c5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bcb767-ddbf7877d-dvwlf" Sep 18 06:53:17.010338 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:17.010287 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw9jd\" (UniqueName: \"kubernetes.io/projected/48708d6d-8b5e-4a16-b3b1-22bf2208d3c5-kube-api-access-tw9jd\") pod \"payload-pre-processing-e2e-api-b-bcb767-ddbf7877d-dvwlf\" (UID: \"48708d6d-8b5e-4a16-b3b1-22bf2208d3c5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bcb767-ddbf7877d-dvwlf" Sep 18 06:53:17.010486 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:17.010343 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0518cd5e-3515-4b8b-8865-c638876b77e2-plugins-config-volume\") pod \"payload-processing-e2e-api-b-bcb767-86b5b575f9-lltb5\" (UID: \"0518cd5e-3515-4b8b-8865-c638876b77e2\") " pod="openshift-ingress/payload-processing-e2e-api-b-bcb767-86b5b575f9-lltb5" Sep 18 06:53:17.110970 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:17.110939 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0518cd5e-3515-4b8b-8865-c638876b77e2-plugins-config-volume\") pod \"payload-processing-e2e-api-b-bcb767-86b5b575f9-lltb5\" (UID: \"0518cd5e-3515-4b8b-8865-c638876b77e2\") " pod="openshift-ingress/payload-processing-e2e-api-b-bcb767-86b5b575f9-lltb5" Sep 18 06:53:17.111133 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:17.110998 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fbjq\" (UniqueName: \"kubernetes.io/projected/0518cd5e-3515-4b8b-8865-c638876b77e2-kube-api-access-8fbjq\") pod \"payload-processing-e2e-api-b-bcb767-86b5b575f9-lltb5\" (UID: \"0518cd5e-3515-4b8b-8865-c638876b77e2\") " pod="openshift-ingress/payload-processing-e2e-api-b-bcb767-86b5b575f9-lltb5" Sep 18 06:53:17.111133 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:17.111022 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48708d6d-8b5e-4a16-b3b1-22bf2208d3c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-bcb767-ddbf7877d-dvwlf\" (UID: \"48708d6d-8b5e-4a16-b3b1-22bf2208d3c5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bcb767-ddbf7877d-dvwlf" Sep 18 06:53:17.111133 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:17.111074 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw9jd\" (UniqueName: \"kubernetes.io/projected/48708d6d-8b5e-4a16-b3b1-22bf2208d3c5-kube-api-access-tw9jd\") pod \"payload-pre-processing-e2e-api-b-bcb767-ddbf7877d-dvwlf\" (UID: \"48708d6d-8b5e-4a16-b3b1-22bf2208d3c5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bcb767-ddbf7877d-dvwlf" Sep 18 06:53:17.111730 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:17.111709 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0518cd5e-3515-4b8b-8865-c638876b77e2-plugins-config-volume\") pod \"payload-processing-e2e-api-b-bcb767-86b5b575f9-lltb5\" (UID: \"0518cd5e-3515-4b8b-8865-c638876b77e2\") " pod="openshift-ingress/payload-processing-e2e-api-b-bcb767-86b5b575f9-lltb5" Sep 18 06:53:17.111787 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:17.111770 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48708d6d-8b5e-4a16-b3b1-22bf2208d3c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-bcb767-ddbf7877d-dvwlf\" (UID: \"48708d6d-8b5e-4a16-b3b1-22bf2208d3c5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bcb767-ddbf7877d-dvwlf" Sep 18 06:53:17.119225 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:17.119199 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw9jd\" (UniqueName: \"kubernetes.io/projected/48708d6d-8b5e-4a16-b3b1-22bf2208d3c5-kube-api-access-tw9jd\") pod \"payload-pre-processing-e2e-api-b-bcb767-ddbf7877d-dvwlf\" (UID: \"48708d6d-8b5e-4a16-b3b1-22bf2208d3c5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bcb767-ddbf7877d-dvwlf" Sep 18 06:53:17.119225 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:17.119213 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fbjq\" (UniqueName: \"kubernetes.io/projected/0518cd5e-3515-4b8b-8865-c638876b77e2-kube-api-access-8fbjq\") pod \"payload-processing-e2e-api-b-bcb767-86b5b575f9-lltb5\" (UID: \"0518cd5e-3515-4b8b-8865-c638876b77e2\") " pod="openshift-ingress/payload-processing-e2e-api-b-bcb767-86b5b575f9-lltb5" Sep 18 06:53:17.241510 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:17.241444 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bcb767-ddbf7877d-dvwlf" Sep 18 06:53:17.274398 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:17.274357 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bcb767-86b5b575f9-lltb5" Sep 18 06:53:17.514233 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:17.514208 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bcb767-ddbf7877d-dvwlf"] Sep 18 06:53:17.515217 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:53:17.515192 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48708d6d_8b5e_4a16_b3b1_22bf2208d3c5.slice/crio-11d9bbfb62798deb4f0a696322540e775e5d3dc6ad4a2819c9619999bd2475d5 WatchSource:0}: Error finding container 11d9bbfb62798deb4f0a696322540e775e5d3dc6ad4a2819c9619999bd2475d5: Status 404 returned error can't find the container with id 11d9bbfb62798deb4f0a696322540e775e5d3dc6ad4a2819c9619999bd2475d5 Sep 18 06:53:17.558399 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:17.558376 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bcb767-86b5b575f9-lltb5"] Sep 18 06:53:17.559693 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:53:17.559668 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0518cd5e_3515_4b8b_8865_c638876b77e2.slice/crio-4e09a53fa07aa5863b314d5c517b79b406922fe85c3af7c59d6fdfbbe450d90e WatchSource:0}: Error finding container 4e09a53fa07aa5863b314d5c517b79b406922fe85c3af7c59d6fdfbbe450d90e: Status 404 returned error can't find the container with id 4e09a53fa07aa5863b314d5c517b79b406922fe85c3af7c59d6fdfbbe450d90e Sep 18 06:53:17.565074 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:17.565047 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bcb767-ddbf7877d-dvwlf" event={"ID":"48708d6d-8b5e-4a16-b3b1-22bf2208d3c5","Type":"ContainerStarted","Data":"11d9bbfb62798deb4f0a696322540e775e5d3dc6ad4a2819c9619999bd2475d5"} Sep 18 06:53:18.575815 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:18.575779 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bcb767-ddbf7877d-dvwlf" event={"ID":"48708d6d-8b5e-4a16-b3b1-22bf2208d3c5","Type":"ContainerStarted","Data":"cce22c885d05e9288749f03444906ef50e4c3f87bc69e5fdc2bef5cf444a8d43"} Sep 18 06:53:18.576277 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:18.575845 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bcb767-ddbf7877d-dvwlf" Sep 18 06:53:18.577658 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:18.577635 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bcb767-86b5b575f9-lltb5" event={"ID":"0518cd5e-3515-4b8b-8865-c638876b77e2","Type":"ContainerStarted","Data":"090b565fe81b7e06ad6024ec2b084082c9954ab2cff2d2191d8dd5cdf2a0903d"} Sep 18 06:53:18.577763 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:18.577672 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bcb767-86b5b575f9-lltb5" event={"ID":"0518cd5e-3515-4b8b-8865-c638876b77e2","Type":"ContainerStarted","Data":"4e09a53fa07aa5863b314d5c517b79b406922fe85c3af7c59d6fdfbbe450d90e"} Sep 18 06:53:18.577763 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:18.577755 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-bcb767-86b5b575f9-lltb5" Sep 18 06:53:18.590627 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:18.590579 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bcb767-ddbf7877d-dvwlf" podStartSLOduration=2.5905629919999997 podStartE2EDuration="2.590562992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:53:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:53:18.589749056 +0000 UTC m=+1860.329165245" watchObservedRunningTime="2026-09-18 06:53:18.590562992 +0000 UTC m=+1860.329979170" Sep 18 06:53:18.604931 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:18.604894 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-bcb767-86b5b575f9-lltb5" podStartSLOduration=2.60488111 podStartE2EDuration="2.60488111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:53:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:53:18.603669281 +0000 UTC m=+1860.343085456" watchObservedRunningTime="2026-09-18 06:53:18.60488111 +0000 UTC m=+1860.344297284" Sep 18 06:53:20.760729 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:20.760700 2567 scope.go:117] "RemoveContainer" containerID="d7d3c4fb9076e5fabc3424ce3833862cc2de77cbaa25fbbc2bf27353e612e166" Sep 18 06:53:29.587771 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:29.587742 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bcb767-ddbf7877d-dvwlf" Sep 18 06:53:29.588272 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:29.587886 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-bcb767-86b5b575f9-lltb5" Sep 18 06:53:44.121928 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:44.121856 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2686de-tkbq5"] Sep 18 06:53:44.124703 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:44.124688 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2686de-tkbq5" Sep 18 06:53:44.127199 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:44.127176 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-88t7h\"" Sep 18 06:53:44.127199 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:44.127192 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 18 06:53:44.132608 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:44.132584 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2686de-tkbq5"] Sep 18 06:53:44.161818 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:44.161794 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd3b3b65-2fec-45e4-8e55-8e99be2f5a12-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2686de-tkbq5\" (UID: \"bd3b3b65-2fec-45e4-8e55-8e99be2f5a12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2686de-tkbq5" Sep 18 06:53:44.263476 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:44.263435 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd3b3b65-2fec-45e4-8e55-8e99be2f5a12-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2686de-tkbq5\" (UID: \"bd3b3b65-2fec-45e4-8e55-8e99be2f5a12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2686de-tkbq5" Sep 18 06:53:44.264297 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:44.264270 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd3b3b65-2fec-45e4-8e55-8e99be2f5a12-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2686de-tkbq5\" (UID: \"bd3b3b65-2fec-45e4-8e55-8e99be2f5a12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2686de-tkbq5" Sep 18 06:53:44.437302 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:44.437225 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2686de-tkbq5" Sep 18 06:53:44.712960 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:44.712933 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2686de-tkbq5"] Sep 18 06:53:44.713817 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:53:44.713792 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd3b3b65_2fec_45e4_8e55_8e99be2f5a12.slice/crio-440ed749b7e89de6fa95c00539f3de571922bc8302d7d0475baabfbe7eb73eaf WatchSource:0}: Error finding container 440ed749b7e89de6fa95c00539f3de571922bc8302d7d0475baabfbe7eb73eaf: Status 404 returned error can't find the container with id 440ed749b7e89de6fa95c00539f3de571922bc8302d7d0475baabfbe7eb73eaf Sep 18 06:53:44.744760 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:44.744733 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2686de-tkbq5" event={"ID":"bd3b3b65-2fec-45e4-8e55-8e99be2f5a12","Type":"ContainerStarted","Data":"440ed749b7e89de6fa95c00539f3de571922bc8302d7d0475baabfbe7eb73eaf"} Sep 18 06:53:45.753784 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:45.753690 2567 generic.go:356] "Generic (PLEG): container finished" podID="bd3b3b65-2fec-45e4-8e55-8e99be2f5a12" containerID="d5d37d5657f3346fcfdd6a6754081c9be8d7fc161c8051d32603f389b3ed99ee" exitCode=0 Sep 18 06:53:45.754145 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:45.753781 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2686de-tkbq5" event={"ID":"bd3b3b65-2fec-45e4-8e55-8e99be2f5a12","Type":"ContainerDied","Data":"d5d37d5657f3346fcfdd6a6754081c9be8d7fc161c8051d32603f389b3ed99ee"} Sep 18 06:53:46.902163 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:46.902140 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2686de-tkbq5" Sep 18 06:53:46.990520 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:46.990486 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd3b3b65-2fec-45e4-8e55-8e99be2f5a12-maas-api-service-ca\") pod \"bd3b3b65-2fec-45e4-8e55-8e99be2f5a12\" (UID: \"bd3b3b65-2fec-45e4-8e55-8e99be2f5a12\") " Sep 18 06:53:46.990972 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:46.990927 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd3b3b65-2fec-45e4-8e55-8e99be2f5a12-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bd3b3b65-2fec-45e4-8e55-8e99be2f5a12" (UID: "bd3b3b65-2fec-45e4-8e55-8e99be2f5a12"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 06:53:46.991344 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:46.991324 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd3b3b65-2fec-45e4-8e55-8e99be2f5a12-maas-api-service-ca\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:53:47.772581 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:47.772533 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2686de-tkbq5" event={"ID":"bd3b3b65-2fec-45e4-8e55-8e99be2f5a12","Type":"ContainerDied","Data":"440ed749b7e89de6fa95c00539f3de571922bc8302d7d0475baabfbe7eb73eaf"} Sep 18 06:53:47.772800 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:47.772602 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="440ed749b7e89de6fa95c00539f3de571922bc8302d7d0475baabfbe7eb73eaf" Sep 18 06:53:47.772800 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:47.772655 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2686de-tkbq5" Sep 18 06:53:55.515206 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:55.515176 2567 patch_prober.go:28] interesting pod/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Sep 18 06:53:55.515557 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:55.515239 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d" podUID="a58e0899-5ec2-4ca5-b517-f3929441a513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Sep 18 06:53:55.839093 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:55.839065 2567 generic.go:356] "Generic (PLEG): container finished" podID="a58e0899-5ec2-4ca5-b517-f3929441a513" containerID="af0536b8581261eb26e1911e939d9aac2ba6cdd4360fd20b9de2da96787220a3" exitCode=0 Sep 18 06:53:55.839278 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:55.839131 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d" event={"ID":"a58e0899-5ec2-4ca5-b517-f3929441a513","Type":"ContainerDied","Data":"af0536b8581261eb26e1911e939d9aac2ba6cdd4360fd20b9de2da96787220a3"} Sep 18 06:53:55.839568 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:55.839549 2567 scope.go:117] "RemoveContainer" containerID="af0536b8581261eb26e1911e939d9aac2ba6cdd4360fd20b9de2da96787220a3" Sep 18 06:53:55.840783 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:55.840747 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:53:55.840893 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:55.840836 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:53:55.840893 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:55.840885 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:53:56.847731 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:56.847689 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d" event={"ID":"a58e0899-5ec2-4ca5-b517-f3929441a513","Type":"ContainerStarted","Data":"5fb495a1cbff199370e995d1953cc85860aa153fd6d7a477ac54292f723bf30a"} Sep 18 06:53:57.203855 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:57.203745 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d" Sep 18 06:53:57.204037 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:57.203994 2567 patch_prober.go:28] interesting pod/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Sep 18 06:53:57.204099 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:57.204050 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d" podUID="a58e0899-5ec2-4ca5-b517-f3929441a513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Sep 18 06:53:58.204494 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:58.204457 2567 patch_prober.go:28] interesting pod/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Sep 18 06:53:58.204886 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:58.204516 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d" podUID="a58e0899-5ec2-4ca5-b517-f3929441a513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Sep 18 06:53:59.203702 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:59.203671 2567 patch_prober.go:28] interesting pod/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Sep 18 06:53:59.203882 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:59.203737 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d" podUID="a58e0899-5ec2-4ca5-b517-f3929441a513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Sep 18 06:53:59.905056 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:59.905018 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2686de-549bdbcdb9-bzgvc"] Sep 18 06:53:59.905622 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:53:59.905555 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-2686de-549bdbcdb9-bzgvc" podUID="5b36a5fd-29ba-48fe-9884-a31a22691fd1" containerName="payload-processing" containerID="cri-o://c5cb7d3f072dacf335c28c8efa45db29909cf7958f6f80f992cef031f4915422" gracePeriod=30 Sep 18 06:54:00.069960 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:00.069896 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2686de-tkbq5"] Sep 18 06:54:00.073991 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:00.073959 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2686de-tkbq5"] Sep 18 06:54:00.204602 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:00.204512 2567 patch_prober.go:28] interesting pod/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Sep 18 06:54:00.204602 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:00.204578 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d" podUID="a58e0899-5ec2-4ca5-b517-f3929441a513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Sep 18 06:54:00.231210 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:00.231171 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2686de-549bdbcdb9-bzgvc" Sep 18 06:54:00.335088 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:00.335053 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8cc8m\" (UniqueName: \"kubernetes.io/projected/5b36a5fd-29ba-48fe-9884-a31a22691fd1-kube-api-access-8cc8m\") pod \"5b36a5fd-29ba-48fe-9884-a31a22691fd1\" (UID: \"5b36a5fd-29ba-48fe-9884-a31a22691fd1\") " Sep 18 06:54:00.335301 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:00.335118 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b36a5fd-29ba-48fe-9884-a31a22691fd1-plugins-config-volume\") pod \"5b36a5fd-29ba-48fe-9884-a31a22691fd1\" (UID: \"5b36a5fd-29ba-48fe-9884-a31a22691fd1\") " Sep 18 06:54:00.335562 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:00.335536 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b36a5fd-29ba-48fe-9884-a31a22691fd1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5b36a5fd-29ba-48fe-9884-a31a22691fd1" (UID: "5b36a5fd-29ba-48fe-9884-a31a22691fd1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 06:54:00.337262 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:00.337234 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b36a5fd-29ba-48fe-9884-a31a22691fd1-kube-api-access-8cc8m" (OuterVolumeSpecName: "kube-api-access-8cc8m") pod "5b36a5fd-29ba-48fe-9884-a31a22691fd1" (UID: "5b36a5fd-29ba-48fe-9884-a31a22691fd1"). InnerVolumeSpecName "kube-api-access-8cc8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 06:54:00.436536 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:00.436499 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cc8m\" (UniqueName: \"kubernetes.io/projected/5b36a5fd-29ba-48fe-9884-a31a22691fd1-kube-api-access-8cc8m\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:54:00.436536 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:00.436529 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b36a5fd-29ba-48fe-9884-a31a22691fd1-plugins-config-volume\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:54:00.448513 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:00.446553 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd3b3b65-2fec-45e4-8e55-8e99be2f5a12" path="/var/lib/kubelet/pods/bd3b3b65-2fec-45e4-8e55-8e99be2f5a12/volumes" Sep 18 06:54:00.906079 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:00.906047 2567 generic.go:356] "Generic (PLEG): container finished" podID="5b36a5fd-29ba-48fe-9884-a31a22691fd1" containerID="c5cb7d3f072dacf335c28c8efa45db29909cf7958f6f80f992cef031f4915422" exitCode=0 Sep 18 06:54:00.906511 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:00.906110 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2686de-549bdbcdb9-bzgvc" Sep 18 06:54:00.906511 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:00.906131 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2686de-549bdbcdb9-bzgvc" event={"ID":"5b36a5fd-29ba-48fe-9884-a31a22691fd1","Type":"ContainerDied","Data":"c5cb7d3f072dacf335c28c8efa45db29909cf7958f6f80f992cef031f4915422"} Sep 18 06:54:00.906511 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:00.906213 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2686de-549bdbcdb9-bzgvc" event={"ID":"5b36a5fd-29ba-48fe-9884-a31a22691fd1","Type":"ContainerDied","Data":"82c58709bc520db4831169e0d329c8f1bddba39cceb3f3ab3ca038f0122661de"} Sep 18 06:54:00.906511 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:00.906234 2567 scope.go:117] "RemoveContainer" containerID="c5cb7d3f072dacf335c28c8efa45db29909cf7958f6f80f992cef031f4915422" Sep 18 06:54:00.916306 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:00.916289 2567 scope.go:117] "RemoveContainer" containerID="c5cb7d3f072dacf335c28c8efa45db29909cf7958f6f80f992cef031f4915422" Sep 18 06:54:00.916594 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:54:00.916571 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5cb7d3f072dacf335c28c8efa45db29909cf7958f6f80f992cef031f4915422\": container with ID starting with c5cb7d3f072dacf335c28c8efa45db29909cf7958f6f80f992cef031f4915422 not found: ID does not exist" containerID="c5cb7d3f072dacf335c28c8efa45db29909cf7958f6f80f992cef031f4915422" Sep 18 06:54:00.916664 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:00.916603 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5cb7d3f072dacf335c28c8efa45db29909cf7958f6f80f992cef031f4915422"} err="failed to get container status \"c5cb7d3f072dacf335c28c8efa45db29909cf7958f6f80f992cef031f4915422\": rpc error: code = NotFound desc = could not find container \"c5cb7d3f072dacf335c28c8efa45db29909cf7958f6f80f992cef031f4915422\": container with ID starting with c5cb7d3f072dacf335c28c8efa45db29909cf7958f6f80f992cef031f4915422 not found: ID does not exist" Sep 18 06:54:00.923418 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:00.923397 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2686de-549bdbcdb9-bzgvc"] Sep 18 06:54:00.926432 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:00.926412 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2686de-549bdbcdb9-bzgvc"] Sep 18 06:54:01.203675 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:01.203590 2567 patch_prober.go:28] interesting pod/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Sep 18 06:54:01.203675 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:01.203651 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d" podUID="a58e0899-5ec2-4ca5-b517-f3929441a513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Sep 18 06:54:02.208467 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:02.208434 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d" Sep 18 06:54:02.208831 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:02.208715 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d" Sep 18 06:54:02.209432 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:02.209414 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d" Sep 18 06:54:02.447705 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:02.447672 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b36a5fd-29ba-48fe-9884-a31a22691fd1" path="/var/lib/kubelet/pods/5b36a5fd-29ba-48fe-9884-a31a22691fd1/volumes" Sep 18 06:54:07.040877 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:07.040841 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9"] Sep 18 06:54:07.041267 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:07.041097 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9" podUID="324189bf-a066-41d2-a81b-ae9df0df7114" containerName="istio-proxy" containerID="cri-o://8221a6a83850acb66995c7cce4ea6ff1dddea04815a7e3645d6b5e91b81980b2" gracePeriod=30 Sep 18 06:54:07.367859 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:07.367828 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bcb767-8mktp"] Sep 18 06:54:07.368340 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:07.368326 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd3b3b65-2fec-45e4-8e55-8e99be2f5a12" containerName="revoke-keys" Sep 18 06:54:07.368400 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:07.368343 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd3b3b65-2fec-45e4-8e55-8e99be2f5a12" containerName="revoke-keys" Sep 18 06:54:07.368400 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:07.368389 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b36a5fd-29ba-48fe-9884-a31a22691fd1" containerName="payload-processing" Sep 18 06:54:07.368400 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:07.368396 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b36a5fd-29ba-48fe-9884-a31a22691fd1" containerName="payload-processing" Sep 18 06:54:07.368518 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:07.368483 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b36a5fd-29ba-48fe-9884-a31a22691fd1" containerName="payload-processing" Sep 18 06:54:07.368518 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:07.368492 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd3b3b65-2fec-45e4-8e55-8e99be2f5a12" containerName="revoke-keys" Sep 18 06:54:07.373000 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:07.372985 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bcb767-8mktp" Sep 18 06:54:07.375871 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:07.375631 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-88t7h\"" Sep 18 06:54:07.375871 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:07.375660 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 18 06:54:07.377692 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:07.377665 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bcb767-8mktp"] Sep 18 06:54:07.508006 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:07.507978 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5adad2f-aeee-4b48-b528-509bdef7be19-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bcb767-8mktp\" (UID: \"b5adad2f-aeee-4b48-b528-509bdef7be19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bcb767-8mktp" Sep 18 06:54:07.609420 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:07.609389 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5adad2f-aeee-4b48-b528-509bdef7be19-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bcb767-8mktp\" (UID: \"b5adad2f-aeee-4b48-b528-509bdef7be19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bcb767-8mktp" Sep 18 06:54:07.609924 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:07.609905 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5adad2f-aeee-4b48-b528-509bdef7be19-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bcb767-8mktp\" (UID: \"b5adad2f-aeee-4b48-b528-509bdef7be19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bcb767-8mktp" Sep 18 06:54:07.686386 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:07.686298 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bcb767-8mktp" Sep 18 06:54:07.959168 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:07.959142 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bcb767-8mktp"] Sep 18 06:54:07.961458 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:54:07.961431 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5adad2f_aeee_4b48_b528_509bdef7be19.slice/crio-90c5b60c0b8998070abce2a231705b994e89c543f074a070a6f4ca845ac13c50 WatchSource:0}: Error finding container 90c5b60c0b8998070abce2a231705b994e89c543f074a070a6f4ca845ac13c50: Status 404 returned error can't find the container with id 90c5b60c0b8998070abce2a231705b994e89c543f074a070a6f4ca845ac13c50 Sep 18 06:54:08.963889 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:08.963812 2567 generic.go:356] "Generic (PLEG): container finished" podID="b5adad2f-aeee-4b48-b528-509bdef7be19" containerID="e78d704256aab946f646b5f6eddc09f959ed0b3a3250f9f4a46fb1ee9b1172f8" exitCode=0 Sep 18 06:54:08.964219 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:08.963891 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bcb767-8mktp" event={"ID":"b5adad2f-aeee-4b48-b528-509bdef7be19","Type":"ContainerDied","Data":"e78d704256aab946f646b5f6eddc09f959ed0b3a3250f9f4a46fb1ee9b1172f8"} Sep 18 06:54:08.964219 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:08.963944 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bcb767-8mktp" event={"ID":"b5adad2f-aeee-4b48-b528-509bdef7be19","Type":"ContainerStarted","Data":"90c5b60c0b8998070abce2a231705b994e89c543f074a070a6f4ca845ac13c50"} Sep 18 06:54:09.782940 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:09.782900 2567 patch_prober.go:28] interesting pod/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 18 06:54:09.783135 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:09.782997 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9" podUID="324189bf-a066-41d2-a81b-ae9df0df7114" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 18 06:54:09.908780 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:09.908748 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-946609-4np7r"] Sep 18 06:54:09.912593 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:09.912573 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-946609-4np7r" Sep 18 06:54:09.919354 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:09.919330 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-946609-4np7r"] Sep 18 06:54:10.035322 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:10.035191 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36181621-0433-4096-8d2a-aa530701b9ad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-946609-4np7r\" (UID: \"36181621-0433-4096-8d2a-aa530701b9ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-946609-4np7r" Sep 18 06:54:10.120868 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:10.120844 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bcb767-8mktp" Sep 18 06:54:10.136243 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:10.136158 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36181621-0433-4096-8d2a-aa530701b9ad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-946609-4np7r\" (UID: \"36181621-0433-4096-8d2a-aa530701b9ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-946609-4np7r" Sep 18 06:54:10.136867 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:10.136840 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36181621-0433-4096-8d2a-aa530701b9ad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-946609-4np7r\" (UID: \"36181621-0433-4096-8d2a-aa530701b9ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-946609-4np7r" Sep 18 06:54:10.225347 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:10.225310 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-946609-4np7r" Sep 18 06:54:10.237635 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:10.237526 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5adad2f-aeee-4b48-b528-509bdef7be19-maas-api-service-ca\") pod \"b5adad2f-aeee-4b48-b528-509bdef7be19\" (UID: \"b5adad2f-aeee-4b48-b528-509bdef7be19\") " Sep 18 06:54:10.237936 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:10.237907 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5adad2f-aeee-4b48-b528-509bdef7be19-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b5adad2f-aeee-4b48-b528-509bdef7be19" (UID: "b5adad2f-aeee-4b48-b528-509bdef7be19"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 06:54:10.338792 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:10.338766 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5adad2f-aeee-4b48-b528-509bdef7be19-maas-api-service-ca\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:54:10.498222 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:10.498181 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-946609-4np7r"] Sep 18 06:54:10.498718 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:54:10.498688 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36181621_0433_4096_8d2a_aa530701b9ad.slice/crio-6708920cac8c23759de43e05aea2c19c6c867d5e7f513687e5526b8bf3e24d37 WatchSource:0}: Error finding container 6708920cac8c23759de43e05aea2c19c6c867d5e7f513687e5526b8bf3e24d37: Status 404 returned error can't find the container with id 6708920cac8c23759de43e05aea2c19c6c867d5e7f513687e5526b8bf3e24d37 Sep 18 06:54:10.982127 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:10.982046 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bcb767-8mktp" Sep 18 06:54:10.982127 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:10.982076 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bcb767-8mktp" event={"ID":"b5adad2f-aeee-4b48-b528-509bdef7be19","Type":"ContainerDied","Data":"90c5b60c0b8998070abce2a231705b994e89c543f074a070a6f4ca845ac13c50"} Sep 18 06:54:10.982127 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:10.982124 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90c5b60c0b8998070abce2a231705b994e89c543f074a070a6f4ca845ac13c50" Sep 18 06:54:10.983783 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:10.983753 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-946609-4np7r" event={"ID":"36181621-0433-4096-8d2a-aa530701b9ad","Type":"ContainerStarted","Data":"21475b459c3abf4277cb10483c8ee028d407ff0fe3350a355ca75bb4fb0c2736"} Sep 18 06:54:10.983907 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:10.983805 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-946609-4np7r" event={"ID":"36181621-0433-4096-8d2a-aa530701b9ad","Type":"ContainerStarted","Data":"6708920cac8c23759de43e05aea2c19c6c867d5e7f513687e5526b8bf3e24d37"} Sep 18 06:54:10.999506 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:10.999459 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-946609-4np7r" podStartSLOduration=1.999445149 podStartE2EDuration="1.999445149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:54:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:54:10.997698177 +0000 UTC m=+1912.737114352" watchObservedRunningTime="2026-09-18 06:54:10.999445149 +0000 UTC m=+1912.738861323" Sep 18 06:54:11.992200 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:11.992163 2567 generic.go:356] "Generic (PLEG): container finished" podID="36181621-0433-4096-8d2a-aa530701b9ad" containerID="21475b459c3abf4277cb10483c8ee028d407ff0fe3350a355ca75bb4fb0c2736" exitCode=0 Sep 18 06:54:11.992597 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:11.992216 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-946609-4np7r" event={"ID":"36181621-0433-4096-8d2a-aa530701b9ad","Type":"ContainerDied","Data":"21475b459c3abf4277cb10483c8ee028d407ff0fe3350a355ca75bb4fb0c2736"} Sep 18 06:54:12.304343 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:12.304320 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9" Sep 18 06:54:12.462569 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:12.462539 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/324189bf-a066-41d2-a81b-ae9df0df7114-istiod-ca-cert\") pod \"324189bf-a066-41d2-a81b-ae9df0df7114\" (UID: \"324189bf-a066-41d2-a81b-ae9df0df7114\") " Sep 18 06:54:12.462732 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:12.462580 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/324189bf-a066-41d2-a81b-ae9df0df7114-istio-podinfo\") pod \"324189bf-a066-41d2-a81b-ae9df0df7114\" (UID: \"324189bf-a066-41d2-a81b-ae9df0df7114\") " Sep 18 06:54:12.462732 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:12.462612 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/324189bf-a066-41d2-a81b-ae9df0df7114-credential-socket\") pod \"324189bf-a066-41d2-a81b-ae9df0df7114\" (UID: \"324189bf-a066-41d2-a81b-ae9df0df7114\") " Sep 18 06:54:12.462732 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:12.462646 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/324189bf-a066-41d2-a81b-ae9df0df7114-workload-certs\") pod \"324189bf-a066-41d2-a81b-ae9df0df7114\" (UID: \"324189bf-a066-41d2-a81b-ae9df0df7114\") " Sep 18 06:54:12.462732 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:12.462669 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/324189bf-a066-41d2-a81b-ae9df0df7114-istio-data\") pod \"324189bf-a066-41d2-a81b-ae9df0df7114\" (UID: \"324189bf-a066-41d2-a81b-ae9df0df7114\") " Sep 18 06:54:12.462732 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:12.462693 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sx4t4\" (UniqueName: \"kubernetes.io/projected/324189bf-a066-41d2-a81b-ae9df0df7114-kube-api-access-sx4t4\") pod \"324189bf-a066-41d2-a81b-ae9df0df7114\" (UID: \"324189bf-a066-41d2-a81b-ae9df0df7114\") " Sep 18 06:54:12.463022 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:12.462813 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/324189bf-a066-41d2-a81b-ae9df0df7114-workload-socket\") pod \"324189bf-a066-41d2-a81b-ae9df0df7114\" (UID: \"324189bf-a066-41d2-a81b-ae9df0df7114\") " Sep 18 06:54:12.463022 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:12.462872 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/324189bf-a066-41d2-a81b-ae9df0df7114-istio-token\") pod \"324189bf-a066-41d2-a81b-ae9df0df7114\" (UID: \"324189bf-a066-41d2-a81b-ae9df0df7114\") " Sep 18 06:54:12.463022 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:12.462901 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/324189bf-a066-41d2-a81b-ae9df0df7114-istio-envoy\") pod \"324189bf-a066-41d2-a81b-ae9df0df7114\" (UID: \"324189bf-a066-41d2-a81b-ae9df0df7114\") " Sep 18 06:54:12.463022 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:12.462928 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/324189bf-a066-41d2-a81b-ae9df0df7114-istio-data" (OuterVolumeSpecName: "istio-data") pod "324189bf-a066-41d2-a81b-ae9df0df7114" (UID: "324189bf-a066-41d2-a81b-ae9df0df7114"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:54:12.463022 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:12.462948 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/324189bf-a066-41d2-a81b-ae9df0df7114-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "324189bf-a066-41d2-a81b-ae9df0df7114" (UID: "324189bf-a066-41d2-a81b-ae9df0df7114"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:54:12.463022 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:12.462938 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/324189bf-a066-41d2-a81b-ae9df0df7114-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "324189bf-a066-41d2-a81b-ae9df0df7114" (UID: "324189bf-a066-41d2-a81b-ae9df0df7114"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:54:12.463022 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:12.462976 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/324189bf-a066-41d2-a81b-ae9df0df7114-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "324189bf-a066-41d2-a81b-ae9df0df7114" (UID: "324189bf-a066-41d2-a81b-ae9df0df7114"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 06:54:12.463357 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:12.463096 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/324189bf-a066-41d2-a81b-ae9df0df7114-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "324189bf-a066-41d2-a81b-ae9df0df7114" (UID: "324189bf-a066-41d2-a81b-ae9df0df7114"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:54:12.463357 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:12.463255 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/324189bf-a066-41d2-a81b-ae9df0df7114-istiod-ca-cert\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:54:12.463357 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:12.463274 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/324189bf-a066-41d2-a81b-ae9df0df7114-credential-socket\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:54:12.463357 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:12.463291 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/324189bf-a066-41d2-a81b-ae9df0df7114-workload-certs\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:54:12.463357 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:12.463305 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/324189bf-a066-41d2-a81b-ae9df0df7114-istio-data\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:54:12.463357 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:12.463317 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/324189bf-a066-41d2-a81b-ae9df0df7114-workload-socket\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:54:12.464929 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:12.464908 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/324189bf-a066-41d2-a81b-ae9df0df7114-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "324189bf-a066-41d2-a81b-ae9df0df7114" (UID: "324189bf-a066-41d2-a81b-ae9df0df7114"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 06:54:12.465046 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:12.465027 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/324189bf-a066-41d2-a81b-ae9df0df7114-kube-api-access-sx4t4" (OuterVolumeSpecName: "kube-api-access-sx4t4") pod "324189bf-a066-41d2-a81b-ae9df0df7114" (UID: "324189bf-a066-41d2-a81b-ae9df0df7114"). InnerVolumeSpecName "kube-api-access-sx4t4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 06:54:12.465315 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:12.465299 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/324189bf-a066-41d2-a81b-ae9df0df7114-istio-token" (OuterVolumeSpecName: "istio-token") pod "324189bf-a066-41d2-a81b-ae9df0df7114" (UID: "324189bf-a066-41d2-a81b-ae9df0df7114"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 06:54:12.465356 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:12.465307 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/324189bf-a066-41d2-a81b-ae9df0df7114-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "324189bf-a066-41d2-a81b-ae9df0df7114" (UID: "324189bf-a066-41d2-a81b-ae9df0df7114"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:54:12.563988 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:12.563930 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/324189bf-a066-41d2-a81b-ae9df0df7114-istio-podinfo\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:54:12.563988 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:12.563954 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx4t4\" (UniqueName: \"kubernetes.io/projected/324189bf-a066-41d2-a81b-ae9df0df7114-kube-api-access-sx4t4\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:54:12.563988 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:12.563965 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/324189bf-a066-41d2-a81b-ae9df0df7114-istio-token\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:54:12.563988 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:12.563973 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/324189bf-a066-41d2-a81b-ae9df0df7114-istio-envoy\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:54:13.006997 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:13.006960 2567 generic.go:356] "Generic (PLEG): container finished" podID="324189bf-a066-41d2-a81b-ae9df0df7114" containerID="8221a6a83850acb66995c7cce4ea6ff1dddea04815a7e3645d6b5e91b81980b2" exitCode=0 Sep 18 06:54:13.007462 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:13.006999 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9" event={"ID":"324189bf-a066-41d2-a81b-ae9df0df7114","Type":"ContainerDied","Data":"8221a6a83850acb66995c7cce4ea6ff1dddea04815a7e3645d6b5e91b81980b2"} Sep 18 06:54:13.007462 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:13.007011 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9" Sep 18 06:54:13.007462 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:13.007098 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9" event={"ID":"324189bf-a066-41d2-a81b-ae9df0df7114","Type":"ContainerDied","Data":"3d788ef8f4ea56d50d2574aa523b819fb29003b9002b1134d9499a7335541006"} Sep 18 06:54:13.007462 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:13.007113 2567 scope.go:117] "RemoveContainer" containerID="8221a6a83850acb66995c7cce4ea6ff1dddea04815a7e3645d6b5e91b81980b2" Sep 18 06:54:13.019019 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:13.019001 2567 scope.go:117] "RemoveContainer" containerID="cd5f3cfe43edb8de043a074c1338febb1af692c93fefb46831b3abcfb260dc48" Sep 18 06:54:13.030378 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:13.030343 2567 scope.go:117] "RemoveContainer" containerID="8221a6a83850acb66995c7cce4ea6ff1dddea04815a7e3645d6b5e91b81980b2" Sep 18 06:54:13.030684 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:54:13.030663 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8221a6a83850acb66995c7cce4ea6ff1dddea04815a7e3645d6b5e91b81980b2\": container with ID starting with 8221a6a83850acb66995c7cce4ea6ff1dddea04815a7e3645d6b5e91b81980b2 not found: ID does not exist" containerID="8221a6a83850acb66995c7cce4ea6ff1dddea04815a7e3645d6b5e91b81980b2" Sep 18 06:54:13.030755 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:13.030694 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8221a6a83850acb66995c7cce4ea6ff1dddea04815a7e3645d6b5e91b81980b2"} err="failed to get container status \"8221a6a83850acb66995c7cce4ea6ff1dddea04815a7e3645d6b5e91b81980b2\": rpc error: code = NotFound desc = could not find container \"8221a6a83850acb66995c7cce4ea6ff1dddea04815a7e3645d6b5e91b81980b2\": container with ID starting with 8221a6a83850acb66995c7cce4ea6ff1dddea04815a7e3645d6b5e91b81980b2 not found: ID does not exist" Sep 18 06:54:13.030755 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:13.030712 2567 scope.go:117] "RemoveContainer" containerID="cd5f3cfe43edb8de043a074c1338febb1af692c93fefb46831b3abcfb260dc48" Sep 18 06:54:13.030994 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:54:13.030975 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd5f3cfe43edb8de043a074c1338febb1af692c93fefb46831b3abcfb260dc48\": container with ID starting with cd5f3cfe43edb8de043a074c1338febb1af692c93fefb46831b3abcfb260dc48 not found: ID does not exist" containerID="cd5f3cfe43edb8de043a074c1338febb1af692c93fefb46831b3abcfb260dc48" Sep 18 06:54:13.031041 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:13.030998 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd5f3cfe43edb8de043a074c1338febb1af692c93fefb46831b3abcfb260dc48"} err="failed to get container status \"cd5f3cfe43edb8de043a074c1338febb1af692c93fefb46831b3abcfb260dc48\": rpc error: code = NotFound desc = could not find container \"cd5f3cfe43edb8de043a074c1338febb1af692c93fefb46831b3abcfb260dc48\": container with ID starting with cd5f3cfe43edb8de043a074c1338febb1af692c93fefb46831b3abcfb260dc48 not found: ID does not exist" Sep 18 06:54:13.033856 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:13.033837 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9"] Sep 18 06:54:13.038800 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:13.038775 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-2686de-openshift-default-78c78f86c8-2g6h9"] Sep 18 06:54:13.152087 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:13.152063 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-946609-4np7r" Sep 18 06:54:13.270240 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:13.270164 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36181621-0433-4096-8d2a-aa530701b9ad-maas-api-service-ca\") pod \"36181621-0433-4096-8d2a-aa530701b9ad\" (UID: \"36181621-0433-4096-8d2a-aa530701b9ad\") " Sep 18 06:54:13.270514 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:13.270494 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36181621-0433-4096-8d2a-aa530701b9ad-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "36181621-0433-4096-8d2a-aa530701b9ad" (UID: "36181621-0433-4096-8d2a-aa530701b9ad"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 06:54:13.371424 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:13.371390 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36181621-0433-4096-8d2a-aa530701b9ad-maas-api-service-ca\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:54:14.015757 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:14.015729 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-946609-4np7r" Sep 18 06:54:14.016163 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:14.015761 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-946609-4np7r" event={"ID":"36181621-0433-4096-8d2a-aa530701b9ad","Type":"ContainerDied","Data":"6708920cac8c23759de43e05aea2c19c6c867d5e7f513687e5526b8bf3e24d37"} Sep 18 06:54:14.016163 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:14.015809 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6708920cac8c23759de43e05aea2c19c6c867d5e7f513687e5526b8bf3e24d37" Sep 18 06:54:14.446064 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:14.446039 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="324189bf-a066-41d2-a81b-ae9df0df7114" path="/var/lib/kubelet/pods/324189bf-a066-41d2-a81b-ae9df0df7114/volumes" Sep 18 06:54:20.784086 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:20.784056 2567 scope.go:117] "RemoveContainer" containerID="875828f6afe6cdaad0997dbe2f492b96dbc7706e0706eaa25504c1092715bb24" Sep 18 06:54:20.793159 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:20.793129 2567 scope.go:117] "RemoveContainer" containerID="11fb717c85d6d7518f03bd27f0e9dcaa06e32d96ea1199a87a5a35288a91b002" Sep 18 06:54:23.123395 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:23.119842 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bcb767-86b5b575f9-lltb5"] Sep 18 06:54:23.123395 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:23.120115 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-bcb767-86b5b575f9-lltb5" podUID="0518cd5e-3515-4b8b-8865-c638876b77e2" containerName="payload-processing" containerID="cri-o://090b565fe81b7e06ad6024ec2b084082c9954ab2cff2d2191d8dd5cdf2a0903d" gracePeriod=30 Sep 18 06:54:23.125314 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:23.125290 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bcb767-ddbf7877d-dvwlf"] Sep 18 06:54:23.126673 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:23.126595 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bcb767-ddbf7877d-dvwlf" podUID="48708d6d-8b5e-4a16-b3b1-22bf2208d3c5" containerName="payload-pre-processing" containerID="cri-o://cce22c885d05e9288749f03444906ef50e4c3f87bc69e5fdc2bef5cf444a8d43" gracePeriod=30 Sep 18 06:54:23.271857 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:23.271773 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bcb767-8mktp"] Sep 18 06:54:23.274802 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:23.274766 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bcb767-8mktp"] Sep 18 06:54:23.454815 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:23.453515 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bcb767-86b5b575f9-lltb5" Sep 18 06:54:23.458824 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:23.458796 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bcb767-ddbf7877d-dvwlf" Sep 18 06:54:23.468415 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:23.468393 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0518cd5e-3515-4b8b-8865-c638876b77e2-plugins-config-volume\") pod \"0518cd5e-3515-4b8b-8865-c638876b77e2\" (UID: \"0518cd5e-3515-4b8b-8865-c638876b77e2\") " Sep 18 06:54:23.468487 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:23.468426 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48708d6d-8b5e-4a16-b3b1-22bf2208d3c5-plugins-config-volume\") pod \"48708d6d-8b5e-4a16-b3b1-22bf2208d3c5\" (UID: \"48708d6d-8b5e-4a16-b3b1-22bf2208d3c5\") " Sep 18 06:54:23.468521 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:23.468485 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8fbjq\" (UniqueName: \"kubernetes.io/projected/0518cd5e-3515-4b8b-8865-c638876b77e2-kube-api-access-8fbjq\") pod \"0518cd5e-3515-4b8b-8865-c638876b77e2\" (UID: \"0518cd5e-3515-4b8b-8865-c638876b77e2\") " Sep 18 06:54:23.468521 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:23.468506 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tw9jd\" (UniqueName: \"kubernetes.io/projected/48708d6d-8b5e-4a16-b3b1-22bf2208d3c5-kube-api-access-tw9jd\") pod \"48708d6d-8b5e-4a16-b3b1-22bf2208d3c5\" (UID: \"48708d6d-8b5e-4a16-b3b1-22bf2208d3c5\") " Sep 18 06:54:23.468830 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:23.468804 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0518cd5e-3515-4b8b-8865-c638876b77e2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0518cd5e-3515-4b8b-8865-c638876b77e2" (UID: "0518cd5e-3515-4b8b-8865-c638876b77e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 06:54:23.468928 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:23.468832 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48708d6d-8b5e-4a16-b3b1-22bf2208d3c5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "48708d6d-8b5e-4a16-b3b1-22bf2208d3c5" (UID: "48708d6d-8b5e-4a16-b3b1-22bf2208d3c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 06:54:23.470781 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:23.470758 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48708d6d-8b5e-4a16-b3b1-22bf2208d3c5-kube-api-access-tw9jd" (OuterVolumeSpecName: "kube-api-access-tw9jd") pod "48708d6d-8b5e-4a16-b3b1-22bf2208d3c5" (UID: "48708d6d-8b5e-4a16-b3b1-22bf2208d3c5"). InnerVolumeSpecName "kube-api-access-tw9jd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 06:54:23.470869 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:23.470835 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0518cd5e-3515-4b8b-8865-c638876b77e2-kube-api-access-8fbjq" (OuterVolumeSpecName: "kube-api-access-8fbjq") pod "0518cd5e-3515-4b8b-8865-c638876b77e2" (UID: "0518cd5e-3515-4b8b-8865-c638876b77e2"). InnerVolumeSpecName "kube-api-access-8fbjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 06:54:23.569528 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:23.569474 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fbjq\" (UniqueName: \"kubernetes.io/projected/0518cd5e-3515-4b8b-8865-c638876b77e2-kube-api-access-8fbjq\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:54:23.569528 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:23.569522 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw9jd\" (UniqueName: \"kubernetes.io/projected/48708d6d-8b5e-4a16-b3b1-22bf2208d3c5-kube-api-access-tw9jd\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:54:23.569528 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:23.569534 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0518cd5e-3515-4b8b-8865-c638876b77e2-plugins-config-volume\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:54:23.569528 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:23.569543 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48708d6d-8b5e-4a16-b3b1-22bf2208d3c5-plugins-config-volume\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:54:24.084168 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:24.084131 2567 generic.go:356] "Generic (PLEG): container finished" podID="48708d6d-8b5e-4a16-b3b1-22bf2208d3c5" containerID="cce22c885d05e9288749f03444906ef50e4c3f87bc69e5fdc2bef5cf444a8d43" exitCode=0 Sep 18 06:54:24.084486 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:24.084176 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bcb767-ddbf7877d-dvwlf" Sep 18 06:54:24.084486 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:24.084209 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bcb767-ddbf7877d-dvwlf" event={"ID":"48708d6d-8b5e-4a16-b3b1-22bf2208d3c5","Type":"ContainerDied","Data":"cce22c885d05e9288749f03444906ef50e4c3f87bc69e5fdc2bef5cf444a8d43"} Sep 18 06:54:24.084486 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:24.084262 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bcb767-ddbf7877d-dvwlf" event={"ID":"48708d6d-8b5e-4a16-b3b1-22bf2208d3c5","Type":"ContainerDied","Data":"11d9bbfb62798deb4f0a696322540e775e5d3dc6ad4a2819c9619999bd2475d5"} Sep 18 06:54:24.084486 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:24.084288 2567 scope.go:117] "RemoveContainer" containerID="cce22c885d05e9288749f03444906ef50e4c3f87bc69e5fdc2bef5cf444a8d43" Sep 18 06:54:24.085590 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:24.085568 2567 generic.go:356] "Generic (PLEG): container finished" podID="0518cd5e-3515-4b8b-8865-c638876b77e2" containerID="090b565fe81b7e06ad6024ec2b084082c9954ab2cff2d2191d8dd5cdf2a0903d" exitCode=0 Sep 18 06:54:24.085693 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:24.085608 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bcb767-86b5b575f9-lltb5" Sep 18 06:54:24.085693 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:24.085648 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bcb767-86b5b575f9-lltb5" event={"ID":"0518cd5e-3515-4b8b-8865-c638876b77e2","Type":"ContainerDied","Data":"090b565fe81b7e06ad6024ec2b084082c9954ab2cff2d2191d8dd5cdf2a0903d"} Sep 18 06:54:24.085824 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:24.085703 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bcb767-86b5b575f9-lltb5" event={"ID":"0518cd5e-3515-4b8b-8865-c638876b77e2","Type":"ContainerDied","Data":"4e09a53fa07aa5863b314d5c517b79b406922fe85c3af7c59d6fdfbbe450d90e"} Sep 18 06:54:24.094694 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:24.094679 2567 scope.go:117] "RemoveContainer" containerID="cce22c885d05e9288749f03444906ef50e4c3f87bc69e5fdc2bef5cf444a8d43" Sep 18 06:54:24.095048 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:54:24.094919 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cce22c885d05e9288749f03444906ef50e4c3f87bc69e5fdc2bef5cf444a8d43\": container with ID starting with cce22c885d05e9288749f03444906ef50e4c3f87bc69e5fdc2bef5cf444a8d43 not found: ID does not exist" containerID="cce22c885d05e9288749f03444906ef50e4c3f87bc69e5fdc2bef5cf444a8d43" Sep 18 06:54:24.095048 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:24.094950 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cce22c885d05e9288749f03444906ef50e4c3f87bc69e5fdc2bef5cf444a8d43"} err="failed to get container status \"cce22c885d05e9288749f03444906ef50e4c3f87bc69e5fdc2bef5cf444a8d43\": rpc error: code = NotFound desc = could not find container \"cce22c885d05e9288749f03444906ef50e4c3f87bc69e5fdc2bef5cf444a8d43\": container with ID starting with cce22c885d05e9288749f03444906ef50e4c3f87bc69e5fdc2bef5cf444a8d43 not found: ID does not exist" Sep 18 06:54:24.095048 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:24.094968 2567 scope.go:117] "RemoveContainer" containerID="090b565fe81b7e06ad6024ec2b084082c9954ab2cff2d2191d8dd5cdf2a0903d" Sep 18 06:54:24.103559 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:24.103545 2567 scope.go:117] "RemoveContainer" containerID="090b565fe81b7e06ad6024ec2b084082c9954ab2cff2d2191d8dd5cdf2a0903d" Sep 18 06:54:24.103802 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:54:24.103780 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"090b565fe81b7e06ad6024ec2b084082c9954ab2cff2d2191d8dd5cdf2a0903d\": container with ID starting with 090b565fe81b7e06ad6024ec2b084082c9954ab2cff2d2191d8dd5cdf2a0903d not found: ID does not exist" containerID="090b565fe81b7e06ad6024ec2b084082c9954ab2cff2d2191d8dd5cdf2a0903d" Sep 18 06:54:24.103853 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:24.103810 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"090b565fe81b7e06ad6024ec2b084082c9954ab2cff2d2191d8dd5cdf2a0903d"} err="failed to get container status \"090b565fe81b7e06ad6024ec2b084082c9954ab2cff2d2191d8dd5cdf2a0903d\": rpc error: code = NotFound desc = could not find container \"090b565fe81b7e06ad6024ec2b084082c9954ab2cff2d2191d8dd5cdf2a0903d\": container with ID starting with 090b565fe81b7e06ad6024ec2b084082c9954ab2cff2d2191d8dd5cdf2a0903d not found: ID does not exist" Sep 18 06:54:24.109254 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:24.109230 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bcb767-ddbf7877d-dvwlf"] Sep 18 06:54:24.113328 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:24.113308 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bcb767-ddbf7877d-dvwlf"] Sep 18 06:54:24.122404 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:24.122379 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bcb767-86b5b575f9-lltb5"] Sep 18 06:54:24.128256 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:24.128235 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bcb767-86b5b575f9-lltb5"] Sep 18 06:54:24.446909 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:24.446832 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0518cd5e-3515-4b8b-8865-c638876b77e2" path="/var/lib/kubelet/pods/0518cd5e-3515-4b8b-8865-c638876b77e2/volumes" Sep 18 06:54:24.447308 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:24.447290 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48708d6d-8b5e-4a16-b3b1-22bf2208d3c5" path="/var/lib/kubelet/pods/48708d6d-8b5e-4a16-b3b1-22bf2208d3c5/volumes" Sep 18 06:54:24.447660 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:24.447647 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5adad2f-aeee-4b48-b528-509bdef7be19" path="/var/lib/kubelet/pods/b5adad2f-aeee-4b48-b528-509bdef7be19/volumes" Sep 18 06:54:27.683555 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:27.683441 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-946609-9fd88897f-68ljl"] Sep 18 06:54:27.684699 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:27.684520 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-946609-9fd88897f-68ljl" podUID="14c2ea04-c360-4273-8aff-ebaadd21df0c" containerName="payload-processing" containerID="cri-o://abb124f0e6e423a13fe2431aed9a450b37670839e966c06494d9396835aaee4d" gracePeriod=30 Sep 18 06:54:27.697892 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:27.697861 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-946609-67f4478978-fmbp7"] Sep 18 06:54:27.698678 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:27.698598 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-946609-67f4478978-fmbp7" podUID="feb1276d-6f0b-4bb0-99f6-d64a45b0a8d0" containerName="payload-pre-processing" containerID="cri-o://e347cf8e4b5645d6bb5127848408b1dc5f06e6abd066066f031cb94cc70a9d0e" gracePeriod=30 Sep 18 06:54:27.826087 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:27.826057 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-946609-4np7r"] Sep 18 06:54:27.829184 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:27.829159 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-946609-4np7r"] Sep 18 06:54:28.007428 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:28.007402 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-946609-9fd88897f-68ljl" Sep 18 06:54:28.010854 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:28.010834 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-946609-67f4478978-fmbp7" Sep 18 06:54:28.113014 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:28.112985 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14c2ea04-c360-4273-8aff-ebaadd21df0c-plugins-config-volume\") pod \"14c2ea04-c360-4273-8aff-ebaadd21df0c\" (UID: \"14c2ea04-c360-4273-8aff-ebaadd21df0c\") " Sep 18 06:54:28.113205 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:28.113076 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4ccc2\" (UniqueName: \"kubernetes.io/projected/14c2ea04-c360-4273-8aff-ebaadd21df0c-kube-api-access-4ccc2\") pod \"14c2ea04-c360-4273-8aff-ebaadd21df0c\" (UID: \"14c2ea04-c360-4273-8aff-ebaadd21df0c\") " Sep 18 06:54:28.113205 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:28.113189 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feb1276d-6f0b-4bb0-99f6-d64a45b0a8d0-plugins-config-volume\") pod \"feb1276d-6f0b-4bb0-99f6-d64a45b0a8d0\" (UID: \"feb1276d-6f0b-4bb0-99f6-d64a45b0a8d0\") " Sep 18 06:54:28.113319 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:28.113274 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vcc7p\" (UniqueName: \"kubernetes.io/projected/feb1276d-6f0b-4bb0-99f6-d64a45b0a8d0-kube-api-access-vcc7p\") pod \"feb1276d-6f0b-4bb0-99f6-d64a45b0a8d0\" (UID: \"feb1276d-6f0b-4bb0-99f6-d64a45b0a8d0\") " Sep 18 06:54:28.113425 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:28.113405 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14c2ea04-c360-4273-8aff-ebaadd21df0c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "14c2ea04-c360-4273-8aff-ebaadd21df0c" (UID: "14c2ea04-c360-4273-8aff-ebaadd21df0c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 06:54:28.113662 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:28.113629 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14c2ea04-c360-4273-8aff-ebaadd21df0c-plugins-config-volume\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:54:28.113662 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:28.113649 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feb1276d-6f0b-4bb0-99f6-d64a45b0a8d0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "feb1276d-6f0b-4bb0-99f6-d64a45b0a8d0" (UID: "feb1276d-6f0b-4bb0-99f6-d64a45b0a8d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 06:54:28.115281 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:28.115257 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14c2ea04-c360-4273-8aff-ebaadd21df0c-kube-api-access-4ccc2" (OuterVolumeSpecName: "kube-api-access-4ccc2") pod "14c2ea04-c360-4273-8aff-ebaadd21df0c" (UID: "14c2ea04-c360-4273-8aff-ebaadd21df0c"). InnerVolumeSpecName "kube-api-access-4ccc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 06:54:28.115545 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:28.115521 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feb1276d-6f0b-4bb0-99f6-d64a45b0a8d0-kube-api-access-vcc7p" (OuterVolumeSpecName: "kube-api-access-vcc7p") pod "feb1276d-6f0b-4bb0-99f6-d64a45b0a8d0" (UID: "feb1276d-6f0b-4bb0-99f6-d64a45b0a8d0"). InnerVolumeSpecName "kube-api-access-vcc7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 06:54:28.118034 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:28.118007 2567 generic.go:356] "Generic (PLEG): container finished" podID="feb1276d-6f0b-4bb0-99f6-d64a45b0a8d0" containerID="e347cf8e4b5645d6bb5127848408b1dc5f06e6abd066066f031cb94cc70a9d0e" exitCode=0 Sep 18 06:54:28.118146 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:28.118054 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-946609-67f4478978-fmbp7" Sep 18 06:54:28.118146 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:28.118088 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-946609-67f4478978-fmbp7" event={"ID":"feb1276d-6f0b-4bb0-99f6-d64a45b0a8d0","Type":"ContainerDied","Data":"e347cf8e4b5645d6bb5127848408b1dc5f06e6abd066066f031cb94cc70a9d0e"} Sep 18 06:54:28.118263 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:28.118150 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-946609-67f4478978-fmbp7" event={"ID":"feb1276d-6f0b-4bb0-99f6-d64a45b0a8d0","Type":"ContainerDied","Data":"6ccf8f33e497812ba3243b0c57b46a7e5e7c44370e70a1341c010fc78b977f80"} Sep 18 06:54:28.118263 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:28.118184 2567 scope.go:117] "RemoveContainer" containerID="e347cf8e4b5645d6bb5127848408b1dc5f06e6abd066066f031cb94cc70a9d0e" Sep 18 06:54:28.121123 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:28.121099 2567 generic.go:356] "Generic (PLEG): container finished" podID="14c2ea04-c360-4273-8aff-ebaadd21df0c" containerID="abb124f0e6e423a13fe2431aed9a450b37670839e966c06494d9396835aaee4d" exitCode=0 Sep 18 06:54:28.121241 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:28.121143 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-946609-9fd88897f-68ljl" Sep 18 06:54:28.121241 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:28.121186 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-946609-9fd88897f-68ljl" event={"ID":"14c2ea04-c360-4273-8aff-ebaadd21df0c","Type":"ContainerDied","Data":"abb124f0e6e423a13fe2431aed9a450b37670839e966c06494d9396835aaee4d"} Sep 18 06:54:28.121241 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:28.121236 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-946609-9fd88897f-68ljl" event={"ID":"14c2ea04-c360-4273-8aff-ebaadd21df0c","Type":"ContainerDied","Data":"0c13f49c7a463edeabd3660619527c8983f3e313050bee4cc3c735fed3f4358b"} Sep 18 06:54:28.139327 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:28.139271 2567 scope.go:117] "RemoveContainer" containerID="e347cf8e4b5645d6bb5127848408b1dc5f06e6abd066066f031cb94cc70a9d0e" Sep 18 06:54:28.139778 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:54:28.139757 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e347cf8e4b5645d6bb5127848408b1dc5f06e6abd066066f031cb94cc70a9d0e\": container with ID starting with e347cf8e4b5645d6bb5127848408b1dc5f06e6abd066066f031cb94cc70a9d0e not found: ID does not exist" containerID="e347cf8e4b5645d6bb5127848408b1dc5f06e6abd066066f031cb94cc70a9d0e" Sep 18 06:54:28.139867 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:28.139790 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e347cf8e4b5645d6bb5127848408b1dc5f06e6abd066066f031cb94cc70a9d0e"} err="failed to get container status \"e347cf8e4b5645d6bb5127848408b1dc5f06e6abd066066f031cb94cc70a9d0e\": rpc error: code = NotFound desc = could not find container \"e347cf8e4b5645d6bb5127848408b1dc5f06e6abd066066f031cb94cc70a9d0e\": container with ID starting with e347cf8e4b5645d6bb5127848408b1dc5f06e6abd066066f031cb94cc70a9d0e not found: ID does not exist" Sep 18 06:54:28.139867 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:28.139810 2567 scope.go:117] "RemoveContainer" containerID="abb124f0e6e423a13fe2431aed9a450b37670839e966c06494d9396835aaee4d" Sep 18 06:54:28.149043 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:28.149021 2567 scope.go:117] "RemoveContainer" containerID="abb124f0e6e423a13fe2431aed9a450b37670839e966c06494d9396835aaee4d" Sep 18 06:54:28.149436 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:54:28.149352 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abb124f0e6e423a13fe2431aed9a450b37670839e966c06494d9396835aaee4d\": container with ID starting with abb124f0e6e423a13fe2431aed9a450b37670839e966c06494d9396835aaee4d not found: ID does not exist" containerID="abb124f0e6e423a13fe2431aed9a450b37670839e966c06494d9396835aaee4d" Sep 18 06:54:28.149436 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:28.149407 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abb124f0e6e423a13fe2431aed9a450b37670839e966c06494d9396835aaee4d"} err="failed to get container status \"abb124f0e6e423a13fe2431aed9a450b37670839e966c06494d9396835aaee4d\": rpc error: code = NotFound desc = could not find container \"abb124f0e6e423a13fe2431aed9a450b37670839e966c06494d9396835aaee4d\": container with ID starting with abb124f0e6e423a13fe2431aed9a450b37670839e966c06494d9396835aaee4d not found: ID does not exist" Sep 18 06:54:28.150470 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:28.150448 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-946609-9fd88897f-68ljl"] Sep 18 06:54:28.153180 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:28.153163 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-946609-9fd88897f-68ljl"] Sep 18 06:54:28.162464 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:28.162444 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-946609-67f4478978-fmbp7"] Sep 18 06:54:28.167852 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:28.167831 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-946609-67f4478978-fmbp7"] Sep 18 06:54:28.214506 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:28.214470 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcc7p\" (UniqueName: \"kubernetes.io/projected/feb1276d-6f0b-4bb0-99f6-d64a45b0a8d0-kube-api-access-vcc7p\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:54:28.214631 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:28.214507 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ccc2\" (UniqueName: \"kubernetes.io/projected/14c2ea04-c360-4273-8aff-ebaadd21df0c-kube-api-access-4ccc2\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:54:28.214631 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:28.214528 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feb1276d-6f0b-4bb0-99f6-d64a45b0a8d0-plugins-config-volume\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:54:28.447249 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:28.447218 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14c2ea04-c360-4273-8aff-ebaadd21df0c" path="/var/lib/kubelet/pods/14c2ea04-c360-4273-8aff-ebaadd21df0c/volumes" Sep 18 06:54:28.447600 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:28.447586 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36181621-0433-4096-8d2a-aa530701b9ad" path="/var/lib/kubelet/pods/36181621-0433-4096-8d2a-aa530701b9ad/volumes" Sep 18 06:54:28.447915 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:28.447903 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="feb1276d-6f0b-4bb0-99f6-d64a45b0a8d0" path="/var/lib/kubelet/pods/feb1276d-6f0b-4bb0-99f6-d64a45b0a8d0/volumes" Sep 18 06:54:30.107041 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:30.106997 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d"] Sep 18 06:54:30.107554 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:30.107209 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d" podUID="a58e0899-5ec2-4ca5-b517-f3929441a513" containerName="istio-proxy" containerID="cri-o://5fb495a1cbff199370e995d1953cc85860aa153fd6d7a477ac54292f723bf30a" gracePeriod=30 Sep 18 06:54:30.437962 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:30.437883 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bac224-pqtsv"] Sep 18 06:54:30.438568 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:30.438548 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48708d6d-8b5e-4a16-b3b1-22bf2208d3c5" containerName="payload-pre-processing" Sep 18 06:54:30.438629 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:30.438573 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="48708d6d-8b5e-4a16-b3b1-22bf2208d3c5" containerName="payload-pre-processing" Sep 18 06:54:30.438629 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:30.438589 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="324189bf-a066-41d2-a81b-ae9df0df7114" containerName="istio-proxy" Sep 18 06:54:30.438629 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:30.438597 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="324189bf-a066-41d2-a81b-ae9df0df7114" containerName="istio-proxy" Sep 18 06:54:30.438629 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:30.438607 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0518cd5e-3515-4b8b-8865-c638876b77e2" containerName="payload-processing" Sep 18 06:54:30.438629 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:30.438617 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="0518cd5e-3515-4b8b-8865-c638876b77e2" containerName="payload-processing" Sep 18 06:54:30.438785 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:30.438633 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="324189bf-a066-41d2-a81b-ae9df0df7114" containerName="istio-proxy" Sep 18 06:54:30.438785 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:30.438643 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="324189bf-a066-41d2-a81b-ae9df0df7114" containerName="istio-proxy" Sep 18 06:54:30.438785 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:30.438655 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14c2ea04-c360-4273-8aff-ebaadd21df0c" containerName="payload-processing" Sep 18 06:54:30.438785 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:30.438663 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="14c2ea04-c360-4273-8aff-ebaadd21df0c" containerName="payload-processing" Sep 18 06:54:30.438785 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:30.438691 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="feb1276d-6f0b-4bb0-99f6-d64a45b0a8d0" containerName="payload-pre-processing" Sep 18 06:54:30.438785 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:30.438700 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="feb1276d-6f0b-4bb0-99f6-d64a45b0a8d0" containerName="payload-pre-processing" Sep 18 06:54:30.438785 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:30.438711 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36181621-0433-4096-8d2a-aa530701b9ad" containerName="revoke-keys" Sep 18 06:54:30.438785 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:30.438719 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="36181621-0433-4096-8d2a-aa530701b9ad" containerName="revoke-keys" Sep 18 06:54:30.438785 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:30.438738 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5adad2f-aeee-4b48-b528-509bdef7be19" containerName="revoke-keys" Sep 18 06:54:30.438785 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:30.438746 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5adad2f-aeee-4b48-b528-509bdef7be19" containerName="revoke-keys" Sep 18 06:54:30.439079 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:30.438856 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="0518cd5e-3515-4b8b-8865-c638876b77e2" containerName="payload-processing" Sep 18 06:54:30.439079 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:30.438868 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="feb1276d-6f0b-4bb0-99f6-d64a45b0a8d0" containerName="payload-pre-processing" Sep 18 06:54:30.439079 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:30.438878 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="36181621-0433-4096-8d2a-aa530701b9ad" containerName="revoke-keys" Sep 18 06:54:30.439079 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:30.438886 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="14c2ea04-c360-4273-8aff-ebaadd21df0c" containerName="payload-processing" Sep 18 06:54:30.439079 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:30.438897 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5adad2f-aeee-4b48-b528-509bdef7be19" containerName="revoke-keys" Sep 18 06:54:30.439079 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:30.438906 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="324189bf-a066-41d2-a81b-ae9df0df7114" containerName="istio-proxy" Sep 18 06:54:30.439079 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:30.438912 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="324189bf-a066-41d2-a81b-ae9df0df7114" containerName="istio-proxy" Sep 18 06:54:30.439079 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:30.438918 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="48708d6d-8b5e-4a16-b3b1-22bf2208d3c5" containerName="payload-pre-processing" Sep 18 06:54:30.443785 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:30.443731 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bac224-pqtsv" Sep 18 06:54:30.446090 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:30.446070 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 18 06:54:30.446422 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:30.446400 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-88t7h\"" Sep 18 06:54:30.453170 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:30.453150 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bac224-pqtsv"] Sep 18 06:54:30.537864 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:30.537830 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8dad4a4-cd2d-4b60-b9ae-20d7d1d821e1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-bac224-pqtsv\" (UID: \"b8dad4a4-cd2d-4b60-b9ae-20d7d1d821e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bac224-pqtsv" Sep 18 06:54:30.638619 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:30.638586 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8dad4a4-cd2d-4b60-b9ae-20d7d1d821e1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-bac224-pqtsv\" (UID: \"b8dad4a4-cd2d-4b60-b9ae-20d7d1d821e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bac224-pqtsv" Sep 18 06:54:30.639159 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:30.639140 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8dad4a4-cd2d-4b60-b9ae-20d7d1d821e1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-bac224-pqtsv\" (UID: \"b8dad4a4-cd2d-4b60-b9ae-20d7d1d821e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bac224-pqtsv" Sep 18 06:54:30.759295 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:30.759221 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bac224-pqtsv" Sep 18 06:54:31.031935 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:31.031907 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bac224-pqtsv"] Sep 18 06:54:31.032709 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:54:31.032677 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8dad4a4_cd2d_4b60_b9ae_20d7d1d821e1.slice/crio-e4587bc183d64c1b91189031864d44e478f90cb30e02b451736f809b92c374d0 WatchSource:0}: Error finding container e4587bc183d64c1b91189031864d44e478f90cb30e02b451736f809b92c374d0: Status 404 returned error can't find the container with id e4587bc183d64c1b91189031864d44e478f90cb30e02b451736f809b92c374d0 Sep 18 06:54:31.143869 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:31.143831 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bac224-pqtsv" event={"ID":"b8dad4a4-cd2d-4b60-b9ae-20d7d1d821e1","Type":"ContainerStarted","Data":"e4587bc183d64c1b91189031864d44e478f90cb30e02b451736f809b92c374d0"} Sep 18 06:54:32.152196 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:32.152156 2567 generic.go:356] "Generic (PLEG): container finished" podID="b8dad4a4-cd2d-4b60-b9ae-20d7d1d821e1" containerID="60b37becd7a6ea35b3cd07ccd807529911f25aaf63f5b7dd704f5ed68cef55a6" exitCode=0 Sep 18 06:54:32.152616 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:32.152249 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bac224-pqtsv" event={"ID":"b8dad4a4-cd2d-4b60-b9ae-20d7d1d821e1","Type":"ContainerDied","Data":"60b37becd7a6ea35b3cd07ccd807529911f25aaf63f5b7dd704f5ed68cef55a6"} Sep 18 06:54:32.211376 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:32.211329 2567 patch_prober.go:28] interesting pod/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 18 06:54:32.211503 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:32.211420 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d" podUID="a58e0899-5ec2-4ca5-b517-f3929441a513" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 18 06:54:33.293622 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:33.293594 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bac224-pqtsv" Sep 18 06:54:33.362845 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:33.362817 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8dad4a4-cd2d-4b60-b9ae-20d7d1d821e1-maas-api-service-ca\") pod \"b8dad4a4-cd2d-4b60-b9ae-20d7d1d821e1\" (UID: \"b8dad4a4-cd2d-4b60-b9ae-20d7d1d821e1\") " Sep 18 06:54:33.363143 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:33.363122 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8dad4a4-cd2d-4b60-b9ae-20d7d1d821e1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b8dad4a4-cd2d-4b60-b9ae-20d7d1d821e1" (UID: "b8dad4a4-cd2d-4b60-b9ae-20d7d1d821e1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 06:54:33.363232 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:33.363213 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8dad4a4-cd2d-4b60-b9ae-20d7d1d821e1-maas-api-service-ca\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:54:34.167295 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:34.167263 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bac224-pqtsv" Sep 18 06:54:34.167485 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:34.167267 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bac224-pqtsv" event={"ID":"b8dad4a4-cd2d-4b60-b9ae-20d7d1d821e1","Type":"ContainerDied","Data":"e4587bc183d64c1b91189031864d44e478f90cb30e02b451736f809b92c374d0"} Sep 18 06:54:34.167485 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:34.167415 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4587bc183d64c1b91189031864d44e478f90cb30e02b451736f809b92c374d0" Sep 18 06:54:35.367266 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:35.367234 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d" Sep 18 06:54:35.486263 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:35.486201 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a58e0899-5ec2-4ca5-b517-f3929441a513-istio-podinfo\") pod \"a58e0899-5ec2-4ca5-b517-f3929441a513\" (UID: \"a58e0899-5ec2-4ca5-b517-f3929441a513\") " Sep 18 06:54:35.486263 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:35.486249 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a58e0899-5ec2-4ca5-b517-f3929441a513-istio-token\") pod \"a58e0899-5ec2-4ca5-b517-f3929441a513\" (UID: \"a58e0899-5ec2-4ca5-b517-f3929441a513\") " Sep 18 06:54:35.486483 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:35.486280 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a58e0899-5ec2-4ca5-b517-f3929441a513-workload-socket\") pod \"a58e0899-5ec2-4ca5-b517-f3929441a513\" (UID: \"a58e0899-5ec2-4ca5-b517-f3929441a513\") " Sep 18 06:54:35.486483 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:35.486323 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a58e0899-5ec2-4ca5-b517-f3929441a513-credential-socket\") pod \"a58e0899-5ec2-4ca5-b517-f3929441a513\" (UID: \"a58e0899-5ec2-4ca5-b517-f3929441a513\") " Sep 18 06:54:35.486483 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:35.486345 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a58e0899-5ec2-4ca5-b517-f3929441a513-istio-envoy\") pod \"a58e0899-5ec2-4ca5-b517-f3929441a513\" (UID: \"a58e0899-5ec2-4ca5-b517-f3929441a513\") " Sep 18 06:54:35.486483 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:35.486386 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a58e0899-5ec2-4ca5-b517-f3929441a513-istiod-ca-cert\") pod \"a58e0899-5ec2-4ca5-b517-f3929441a513\" (UID: \"a58e0899-5ec2-4ca5-b517-f3929441a513\") " Sep 18 06:54:35.486483 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:35.486449 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vf285\" (UniqueName: \"kubernetes.io/projected/a58e0899-5ec2-4ca5-b517-f3929441a513-kube-api-access-vf285\") pod \"a58e0899-5ec2-4ca5-b517-f3929441a513\" (UID: \"a58e0899-5ec2-4ca5-b517-f3929441a513\") " Sep 18 06:54:35.486727 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:35.486507 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a58e0899-5ec2-4ca5-b517-f3929441a513-workload-certs\") pod \"a58e0899-5ec2-4ca5-b517-f3929441a513\" (UID: \"a58e0899-5ec2-4ca5-b517-f3929441a513\") " Sep 18 06:54:35.486727 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:35.486540 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a58e0899-5ec2-4ca5-b517-f3929441a513-istio-data\") pod \"a58e0899-5ec2-4ca5-b517-f3929441a513\" (UID: \"a58e0899-5ec2-4ca5-b517-f3929441a513\") " Sep 18 06:54:35.486727 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:35.486716 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a58e0899-5ec2-4ca5-b517-f3929441a513-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a58e0899-5ec2-4ca5-b517-f3929441a513" (UID: "a58e0899-5ec2-4ca5-b517-f3929441a513"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:54:35.486875 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:35.486740 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a58e0899-5ec2-4ca5-b517-f3929441a513-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a58e0899-5ec2-4ca5-b517-f3929441a513" (UID: "a58e0899-5ec2-4ca5-b517-f3929441a513"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:54:35.486928 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:35.486891 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a58e0899-5ec2-4ca5-b517-f3929441a513-istio-data" (OuterVolumeSpecName: "istio-data") pod "a58e0899-5ec2-4ca5-b517-f3929441a513" (UID: "a58e0899-5ec2-4ca5-b517-f3929441a513"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:54:35.487122 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:35.487097 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a58e0899-5ec2-4ca5-b517-f3929441a513-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a58e0899-5ec2-4ca5-b517-f3929441a513" (UID: "a58e0899-5ec2-4ca5-b517-f3929441a513"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:54:35.487219 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:35.487125 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a58e0899-5ec2-4ca5-b517-f3929441a513-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a58e0899-5ec2-4ca5-b517-f3929441a513" (UID: "a58e0899-5ec2-4ca5-b517-f3929441a513"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 06:54:35.487290 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:35.487241 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a58e0899-5ec2-4ca5-b517-f3929441a513-workload-socket\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:54:35.487290 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:35.487265 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a58e0899-5ec2-4ca5-b517-f3929441a513-credential-socket\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:54:35.487290 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:35.487279 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a58e0899-5ec2-4ca5-b517-f3929441a513-istio-data\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:54:35.488608 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:35.488586 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a58e0899-5ec2-4ca5-b517-f3929441a513-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a58e0899-5ec2-4ca5-b517-f3929441a513" (UID: "a58e0899-5ec2-4ca5-b517-f3929441a513"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:54:35.488808 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:35.488794 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a58e0899-5ec2-4ca5-b517-f3929441a513-istio-token" (OuterVolumeSpecName: "istio-token") pod "a58e0899-5ec2-4ca5-b517-f3929441a513" (UID: "a58e0899-5ec2-4ca5-b517-f3929441a513"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 06:54:35.488889 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:35.488871 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a58e0899-5ec2-4ca5-b517-f3929441a513-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a58e0899-5ec2-4ca5-b517-f3929441a513" (UID: "a58e0899-5ec2-4ca5-b517-f3929441a513"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 06:54:35.489117 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:35.489098 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a58e0899-5ec2-4ca5-b517-f3929441a513-kube-api-access-vf285" (OuterVolumeSpecName: "kube-api-access-vf285") pod "a58e0899-5ec2-4ca5-b517-f3929441a513" (UID: "a58e0899-5ec2-4ca5-b517-f3929441a513"). InnerVolumeSpecName "kube-api-access-vf285". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 06:54:35.588255 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:35.588231 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf285\" (UniqueName: \"kubernetes.io/projected/a58e0899-5ec2-4ca5-b517-f3929441a513-kube-api-access-vf285\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:54:35.588255 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:35.588252 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a58e0899-5ec2-4ca5-b517-f3929441a513-workload-certs\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:54:35.588417 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:35.588262 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a58e0899-5ec2-4ca5-b517-f3929441a513-istio-podinfo\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:54:35.588417 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:35.588270 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a58e0899-5ec2-4ca5-b517-f3929441a513-istio-token\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:54:35.588417 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:35.588279 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a58e0899-5ec2-4ca5-b517-f3929441a513-istio-envoy\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:54:35.588417 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:35.588286 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a58e0899-5ec2-4ca5-b517-f3929441a513-istiod-ca-cert\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:54:36.181957 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:36.181923 2567 generic.go:356] "Generic (PLEG): container finished" podID="a58e0899-5ec2-4ca5-b517-f3929441a513" containerID="5fb495a1cbff199370e995d1953cc85860aa153fd6d7a477ac54292f723bf30a" exitCode=0 Sep 18 06:54:36.182130 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:36.181965 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d" Sep 18 06:54:36.182130 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:36.181996 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d" event={"ID":"a58e0899-5ec2-4ca5-b517-f3929441a513","Type":"ContainerDied","Data":"5fb495a1cbff199370e995d1953cc85860aa153fd6d7a477ac54292f723bf30a"} Sep 18 06:54:36.182130 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:36.182053 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d" event={"ID":"a58e0899-5ec2-4ca5-b517-f3929441a513","Type":"ContainerDied","Data":"d6fc4fc93d6fd1406a8c762b5ac5b9025dd7de0d54c1e1d70a50b553318bc2c8"} Sep 18 06:54:36.182130 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:36.182077 2567 scope.go:117] "RemoveContainer" containerID="5fb495a1cbff199370e995d1953cc85860aa153fd6d7a477ac54292f723bf30a" Sep 18 06:54:36.198513 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:36.198495 2567 scope.go:117] "RemoveContainer" containerID="af0536b8581261eb26e1911e939d9aac2ba6cdd4360fd20b9de2da96787220a3" Sep 18 06:54:36.207517 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:36.207499 2567 scope.go:117] "RemoveContainer" containerID="5fb495a1cbff199370e995d1953cc85860aa153fd6d7a477ac54292f723bf30a" Sep 18 06:54:36.207805 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:54:36.207786 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fb495a1cbff199370e995d1953cc85860aa153fd6d7a477ac54292f723bf30a\": container with ID starting with 5fb495a1cbff199370e995d1953cc85860aa153fd6d7a477ac54292f723bf30a not found: ID does not exist" containerID="5fb495a1cbff199370e995d1953cc85860aa153fd6d7a477ac54292f723bf30a" Sep 18 06:54:36.207867 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:36.207812 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fb495a1cbff199370e995d1953cc85860aa153fd6d7a477ac54292f723bf30a"} err="failed to get container status \"5fb495a1cbff199370e995d1953cc85860aa153fd6d7a477ac54292f723bf30a\": rpc error: code = NotFound desc = could not find container \"5fb495a1cbff199370e995d1953cc85860aa153fd6d7a477ac54292f723bf30a\": container with ID starting with 5fb495a1cbff199370e995d1953cc85860aa153fd6d7a477ac54292f723bf30a not found: ID does not exist" Sep 18 06:54:36.207867 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:36.207827 2567 scope.go:117] "RemoveContainer" containerID="af0536b8581261eb26e1911e939d9aac2ba6cdd4360fd20b9de2da96787220a3" Sep 18 06:54:36.208077 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:54:36.208053 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af0536b8581261eb26e1911e939d9aac2ba6cdd4360fd20b9de2da96787220a3\": container with ID starting with af0536b8581261eb26e1911e939d9aac2ba6cdd4360fd20b9de2da96787220a3 not found: ID does not exist" containerID="af0536b8581261eb26e1911e939d9aac2ba6cdd4360fd20b9de2da96787220a3" Sep 18 06:54:36.208186 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:36.208118 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af0536b8581261eb26e1911e939d9aac2ba6cdd4360fd20b9de2da96787220a3"} err="failed to get container status \"af0536b8581261eb26e1911e939d9aac2ba6cdd4360fd20b9de2da96787220a3\": rpc error: code = NotFound desc = could not find container \"af0536b8581261eb26e1911e939d9aac2ba6cdd4360fd20b9de2da96787220a3\": container with ID starting with af0536b8581261eb26e1911e939d9aac2ba6cdd4360fd20b9de2da96787220a3 not found: ID does not exist" Sep 18 06:54:36.208698 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:36.208665 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d"] Sep 18 06:54:36.211734 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:36.211713 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-bcb767-openshift-default-7d6844fdb4-qv48d"] Sep 18 06:54:36.447294 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:36.447216 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a58e0899-5ec2-4ca5-b517-f3929441a513" path="/var/lib/kubelet/pods/a58e0899-5ec2-4ca5-b517-f3929441a513/volumes" Sep 18 06:54:41.180766 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:41.180728 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-bac224-58676c465c-2j9m2"] Sep 18 06:54:41.181189 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:41.181002 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bac224-58676c465c-2j9m2" podUID="53043d19-d768-4926-9193-ba48cf5508da" containerName="payload-processing" containerID="cri-o://dd8e92711717fb7cf548dfbae80d21d9faaa21fa69e36c8d93743b15e1b2f1f0" gracePeriod=30 Sep 18 06:54:41.187051 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:41.187024 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bac224-6b5977dbc4-wrfrp"] Sep 18 06:54:41.215026 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:41.214982 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bac224-6b5977dbc4-wrfrp" podUID="3130492d-c8d9-453b-bd48-13422815eb42" containerName="payload-pre-processing" containerID="cri-o://feeedee4db5197541329f44c1a29656853240aa9331fd0f0e8d7048569ccc4dc" gracePeriod=30 Sep 18 06:54:41.323979 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:41.323950 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bac224-pqtsv"] Sep 18 06:54:41.327408 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:41.327382 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bac224-pqtsv"] Sep 18 06:54:41.509835 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:41.509807 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bac224-58676c465c-2j9m2" Sep 18 06:54:41.513687 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:41.513660 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bac224-6b5977dbc4-wrfrp" Sep 18 06:54:41.647888 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:41.647853 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53043d19-d768-4926-9193-ba48cf5508da-plugins-config-volume\") pod \"53043d19-d768-4926-9193-ba48cf5508da\" (UID: \"53043d19-d768-4926-9193-ba48cf5508da\") " Sep 18 06:54:41.647888 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:41.647894 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fshl5\" (UniqueName: \"kubernetes.io/projected/3130492d-c8d9-453b-bd48-13422815eb42-kube-api-access-fshl5\") pod \"3130492d-c8d9-453b-bd48-13422815eb42\" (UID: \"3130492d-c8d9-453b-bd48-13422815eb42\") " Sep 18 06:54:41.648087 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:41.647928 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3130492d-c8d9-453b-bd48-13422815eb42-plugins-config-volume\") pod \"3130492d-c8d9-453b-bd48-13422815eb42\" (UID: \"3130492d-c8d9-453b-bd48-13422815eb42\") " Sep 18 06:54:41.648087 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:41.647992 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p7gdt\" (UniqueName: \"kubernetes.io/projected/53043d19-d768-4926-9193-ba48cf5508da-kube-api-access-p7gdt\") pod \"53043d19-d768-4926-9193-ba48cf5508da\" (UID: \"53043d19-d768-4926-9193-ba48cf5508da\") " Sep 18 06:54:41.648417 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:41.648344 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53043d19-d768-4926-9193-ba48cf5508da-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "53043d19-d768-4926-9193-ba48cf5508da" (UID: "53043d19-d768-4926-9193-ba48cf5508da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 06:54:41.648417 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:41.648359 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3130492d-c8d9-453b-bd48-13422815eb42-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3130492d-c8d9-453b-bd48-13422815eb42" (UID: "3130492d-c8d9-453b-bd48-13422815eb42"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 06:54:41.650100 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:41.650075 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3130492d-c8d9-453b-bd48-13422815eb42-kube-api-access-fshl5" (OuterVolumeSpecName: "kube-api-access-fshl5") pod "3130492d-c8d9-453b-bd48-13422815eb42" (UID: "3130492d-c8d9-453b-bd48-13422815eb42"). InnerVolumeSpecName "kube-api-access-fshl5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 06:54:41.650176 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:41.650152 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53043d19-d768-4926-9193-ba48cf5508da-kube-api-access-p7gdt" (OuterVolumeSpecName: "kube-api-access-p7gdt") pod "53043d19-d768-4926-9193-ba48cf5508da" (UID: "53043d19-d768-4926-9193-ba48cf5508da"). InnerVolumeSpecName "kube-api-access-p7gdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 06:54:41.749666 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:41.749587 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7gdt\" (UniqueName: \"kubernetes.io/projected/53043d19-d768-4926-9193-ba48cf5508da-kube-api-access-p7gdt\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:54:41.749666 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:41.749614 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53043d19-d768-4926-9193-ba48cf5508da-plugins-config-volume\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:54:41.749666 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:41.749623 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fshl5\" (UniqueName: \"kubernetes.io/projected/3130492d-c8d9-453b-bd48-13422815eb42-kube-api-access-fshl5\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:54:41.749666 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:41.749631 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3130492d-c8d9-453b-bd48-13422815eb42-plugins-config-volume\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:54:42.223085 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:42.223054 2567 generic.go:356] "Generic (PLEG): container finished" podID="53043d19-d768-4926-9193-ba48cf5508da" containerID="dd8e92711717fb7cf548dfbae80d21d9faaa21fa69e36c8d93743b15e1b2f1f0" exitCode=0 Sep 18 06:54:42.223544 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:42.223104 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bac224-58676c465c-2j9m2" Sep 18 06:54:42.223544 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:42.223130 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bac224-58676c465c-2j9m2" event={"ID":"53043d19-d768-4926-9193-ba48cf5508da","Type":"ContainerDied","Data":"dd8e92711717fb7cf548dfbae80d21d9faaa21fa69e36c8d93743b15e1b2f1f0"} Sep 18 06:54:42.223544 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:42.223172 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bac224-58676c465c-2j9m2" event={"ID":"53043d19-d768-4926-9193-ba48cf5508da","Type":"ContainerDied","Data":"4c316aebb3b67c1ea0ab301ab32afdfe84982f05c6bfcdc0e4d87dd1ed8cdc7f"} Sep 18 06:54:42.223544 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:42.223194 2567 scope.go:117] "RemoveContainer" containerID="dd8e92711717fb7cf548dfbae80d21d9faaa21fa69e36c8d93743b15e1b2f1f0" Sep 18 06:54:42.224566 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:42.224483 2567 generic.go:356] "Generic (PLEG): container finished" podID="3130492d-c8d9-453b-bd48-13422815eb42" containerID="feeedee4db5197541329f44c1a29656853240aa9331fd0f0e8d7048569ccc4dc" exitCode=0 Sep 18 06:54:42.224566 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:42.224521 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bac224-6b5977dbc4-wrfrp" event={"ID":"3130492d-c8d9-453b-bd48-13422815eb42","Type":"ContainerDied","Data":"feeedee4db5197541329f44c1a29656853240aa9331fd0f0e8d7048569ccc4dc"} Sep 18 06:54:42.224566 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:42.224539 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bac224-6b5977dbc4-wrfrp" Sep 18 06:54:42.224709 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:42.224572 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bac224-6b5977dbc4-wrfrp" event={"ID":"3130492d-c8d9-453b-bd48-13422815eb42","Type":"ContainerDied","Data":"6e3534a4d3ac802d3e375eaf6b636c65cbb7d704758f1f2e5a5301700aa9a720"} Sep 18 06:54:42.233346 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:42.233330 2567 scope.go:117] "RemoveContainer" containerID="dd8e92711717fb7cf548dfbae80d21d9faaa21fa69e36c8d93743b15e1b2f1f0" Sep 18 06:54:42.233694 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:54:42.233672 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd8e92711717fb7cf548dfbae80d21d9faaa21fa69e36c8d93743b15e1b2f1f0\": container with ID starting with dd8e92711717fb7cf548dfbae80d21d9faaa21fa69e36c8d93743b15e1b2f1f0 not found: ID does not exist" containerID="dd8e92711717fb7cf548dfbae80d21d9faaa21fa69e36c8d93743b15e1b2f1f0" Sep 18 06:54:42.233789 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:42.233704 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd8e92711717fb7cf548dfbae80d21d9faaa21fa69e36c8d93743b15e1b2f1f0"} err="failed to get container status \"dd8e92711717fb7cf548dfbae80d21d9faaa21fa69e36c8d93743b15e1b2f1f0\": rpc error: code = NotFound desc = could not find container \"dd8e92711717fb7cf548dfbae80d21d9faaa21fa69e36c8d93743b15e1b2f1f0\": container with ID starting with dd8e92711717fb7cf548dfbae80d21d9faaa21fa69e36c8d93743b15e1b2f1f0 not found: ID does not exist" Sep 18 06:54:42.233789 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:42.233725 2567 scope.go:117] "RemoveContainer" containerID="feeedee4db5197541329f44c1a29656853240aa9331fd0f0e8d7048569ccc4dc" Sep 18 06:54:42.242455 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:42.242440 2567 scope.go:117] "RemoveContainer" containerID="feeedee4db5197541329f44c1a29656853240aa9331fd0f0e8d7048569ccc4dc" Sep 18 06:54:42.242728 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:54:42.242709 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"feeedee4db5197541329f44c1a29656853240aa9331fd0f0e8d7048569ccc4dc\": container with ID starting with feeedee4db5197541329f44c1a29656853240aa9331fd0f0e8d7048569ccc4dc not found: ID does not exist" containerID="feeedee4db5197541329f44c1a29656853240aa9331fd0f0e8d7048569ccc4dc" Sep 18 06:54:42.242782 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:42.242734 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"feeedee4db5197541329f44c1a29656853240aa9331fd0f0e8d7048569ccc4dc"} err="failed to get container status \"feeedee4db5197541329f44c1a29656853240aa9331fd0f0e8d7048569ccc4dc\": rpc error: code = NotFound desc = could not find container \"feeedee4db5197541329f44c1a29656853240aa9331fd0f0e8d7048569ccc4dc\": container with ID starting with feeedee4db5197541329f44c1a29656853240aa9331fd0f0e8d7048569ccc4dc not found: ID does not exist" Sep 18 06:54:42.253114 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:42.253086 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bac224-6b5977dbc4-wrfrp"] Sep 18 06:54:42.263237 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:42.263217 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bac224-6b5977dbc4-wrfrp"] Sep 18 06:54:42.271577 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:42.271558 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-bac224-58676c465c-2j9m2"] Sep 18 06:54:42.274554 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:42.274529 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-bac224-58676c465c-2j9m2"] Sep 18 06:54:42.446335 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:42.446306 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3130492d-c8d9-453b-bd48-13422815eb42" path="/var/lib/kubelet/pods/3130492d-c8d9-453b-bd48-13422815eb42/volumes" Sep 18 06:54:42.446732 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:42.446717 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53043d19-d768-4926-9193-ba48cf5508da" path="/var/lib/kubelet/pods/53043d19-d768-4926-9193-ba48cf5508da/volumes" Sep 18 06:54:42.447077 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:42.447064 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8dad4a4-cd2d-4b60-b9ae-20d7d1d821e1" path="/var/lib/kubelet/pods/b8dad4a4-cd2d-4b60-b9ae-20d7d1d821e1/volumes" Sep 18 06:54:48.056075 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:48.056044 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp"] Sep 18 06:54:48.056561 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:48.056249 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp" podUID="1d74f3c3-8297-4e77-9223-5bba5a963fea" containerName="istio-proxy" containerID="cri-o://593f1f924f7fc815fc1122f6c46f87bc05280aed4f7c591ec72fcdbec86a7b26" gracePeriod=30 Sep 18 06:54:48.395754 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:48.395725 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-87d618-vbgxd"] Sep 18 06:54:48.396168 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:48.396156 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a58e0899-5ec2-4ca5-b517-f3929441a513" containerName="istio-proxy" Sep 18 06:54:48.396215 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:48.396170 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a58e0899-5ec2-4ca5-b517-f3929441a513" containerName="istio-proxy" Sep 18 06:54:48.396215 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:48.396180 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3130492d-c8d9-453b-bd48-13422815eb42" containerName="payload-pre-processing" Sep 18 06:54:48.396215 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:48.396188 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3130492d-c8d9-453b-bd48-13422815eb42" containerName="payload-pre-processing" Sep 18 06:54:48.396215 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:48.396198 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a58e0899-5ec2-4ca5-b517-f3929441a513" containerName="istio-proxy" Sep 18 06:54:48.396215 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:48.396204 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a58e0899-5ec2-4ca5-b517-f3929441a513" containerName="istio-proxy" Sep 18 06:54:48.396215 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:48.396214 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8dad4a4-cd2d-4b60-b9ae-20d7d1d821e1" containerName="revoke-keys" Sep 18 06:54:48.396432 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:48.396219 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8dad4a4-cd2d-4b60-b9ae-20d7d1d821e1" containerName="revoke-keys" Sep 18 06:54:48.396432 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:48.396229 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53043d19-d768-4926-9193-ba48cf5508da" containerName="payload-processing" Sep 18 06:54:48.396432 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:48.396235 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="53043d19-d768-4926-9193-ba48cf5508da" containerName="payload-processing" Sep 18 06:54:48.396432 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:48.396300 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="53043d19-d768-4926-9193-ba48cf5508da" containerName="payload-processing" Sep 18 06:54:48.396432 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:48.396310 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a58e0899-5ec2-4ca5-b517-f3929441a513" containerName="istio-proxy" Sep 18 06:54:48.396432 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:48.396317 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="3130492d-c8d9-453b-bd48-13422815eb42" containerName="payload-pre-processing" Sep 18 06:54:48.396432 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:48.396322 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8dad4a4-cd2d-4b60-b9ae-20d7d1d821e1" containerName="revoke-keys" Sep 18 06:54:48.396432 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:48.396328 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a58e0899-5ec2-4ca5-b517-f3929441a513" containerName="istio-proxy" Sep 18 06:54:48.399451 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:48.399435 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-87d618-vbgxd" Sep 18 06:54:48.402335 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:48.402311 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 18 06:54:48.402335 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:48.402344 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-88t7h\"" Sep 18 06:54:48.406360 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:48.406302 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-87d618-vbgxd"] Sep 18 06:54:48.517743 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:48.517710 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a75c28e0-bbea-4757-95f6-13639e62aba4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-87d618-vbgxd\" (UID: \"a75c28e0-bbea-4757-95f6-13639e62aba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-87d618-vbgxd" Sep 18 06:54:48.618265 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:48.618236 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a75c28e0-bbea-4757-95f6-13639e62aba4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-87d618-vbgxd\" (UID: \"a75c28e0-bbea-4757-95f6-13639e62aba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-87d618-vbgxd" Sep 18 06:54:48.618857 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:48.618837 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a75c28e0-bbea-4757-95f6-13639e62aba4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-87d618-vbgxd\" (UID: \"a75c28e0-bbea-4757-95f6-13639e62aba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-87d618-vbgxd" Sep 18 06:54:48.713698 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:48.713632 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-87d618-vbgxd" Sep 18 06:54:48.986583 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:48.986551 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-87d618-vbgxd"] Sep 18 06:54:48.987759 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:54:48.987730 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda75c28e0_bbea_4757_95f6_13639e62aba4.slice/crio-cb7d9b3a8187574c739577a5336a6b860ce571f297ce2112f243e35cdbfbcbe1 WatchSource:0}: Error finding container cb7d9b3a8187574c739577a5336a6b860ce571f297ce2112f243e35cdbfbcbe1: Status 404 returned error can't find the container with id cb7d9b3a8187574c739577a5336a6b860ce571f297ce2112f243e35cdbfbcbe1 Sep 18 06:54:49.278939 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:49.278912 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-87d618-vbgxd" event={"ID":"a75c28e0-bbea-4757-95f6-13639e62aba4","Type":"ContainerStarted","Data":"4b9977a109b5f49fe314a8c0c62d379e672b5b5e74a7b9c278f19b867a82140d"} Sep 18 06:54:49.279298 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:49.278961 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-87d618-vbgxd" event={"ID":"a75c28e0-bbea-4757-95f6-13639e62aba4","Type":"ContainerStarted","Data":"cb7d9b3a8187574c739577a5336a6b860ce571f297ce2112f243e35cdbfbcbe1"} Sep 18 06:54:49.293272 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:49.293225 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-87d618-vbgxd" podStartSLOduration=1.2932109760000001 podStartE2EDuration="1.293210976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:54:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:54:49.291819201 +0000 UTC m=+1951.031235398" watchObservedRunningTime="2026-09-18 06:54:49.293210976 +0000 UTC m=+1951.032627147" Sep 18 06:54:49.959082 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:49.959009 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-654463-29828565-zkvz8"] Sep 18 06:54:49.961687 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:49.961664 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-654463-29828565-zkvz8"] Sep 18 06:54:49.978349 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:49.978309 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-654463-db8fb8547-x6klj"] Sep 18 06:54:49.978604 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:49.978574 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-654463-db8fb8547-x6klj" podUID="885403b9-b7b0-4cc0-aa50-9486ba22a99d" containerName="payload-processing" containerID="cri-o://79ee1af26893305254fac413f967d0b1c14740c99c82ae74994cd1e148e265b3" gracePeriod=30 Sep 18 06:54:50.270535 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:50.270508 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-654463-db8fb8547-x6klj" Sep 18 06:54:50.287138 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:50.287108 2567 generic.go:356] "Generic (PLEG): container finished" podID="885403b9-b7b0-4cc0-aa50-9486ba22a99d" containerID="79ee1af26893305254fac413f967d0b1c14740c99c82ae74994cd1e148e265b3" exitCode=0 Sep 18 06:54:50.287524 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:50.287142 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-654463-db8fb8547-x6klj" Sep 18 06:54:50.287524 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:50.287192 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-654463-db8fb8547-x6klj" event={"ID":"885403b9-b7b0-4cc0-aa50-9486ba22a99d","Type":"ContainerDied","Data":"79ee1af26893305254fac413f967d0b1c14740c99c82ae74994cd1e148e265b3"} Sep 18 06:54:50.287524 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:50.287240 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-654463-db8fb8547-x6klj" event={"ID":"885403b9-b7b0-4cc0-aa50-9486ba22a99d","Type":"ContainerDied","Data":"2a8667755c64de5d27eb324c9afd42731c547025b824b24fb2d19979ab0a2e3c"} Sep 18 06:54:50.287524 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:50.287262 2567 scope.go:117] "RemoveContainer" containerID="79ee1af26893305254fac413f967d0b1c14740c99c82ae74994cd1e148e265b3" Sep 18 06:54:50.290403 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:50.290351 2567 generic.go:356] "Generic (PLEG): container finished" podID="a75c28e0-bbea-4757-95f6-13639e62aba4" containerID="4b9977a109b5f49fe314a8c0c62d379e672b5b5e74a7b9c278f19b867a82140d" exitCode=0 Sep 18 06:54:50.290872 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:50.290443 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-87d618-vbgxd" event={"ID":"a75c28e0-bbea-4757-95f6-13639e62aba4","Type":"ContainerDied","Data":"4b9977a109b5f49fe314a8c0c62d379e672b5b5e74a7b9c278f19b867a82140d"} Sep 18 06:54:50.302646 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:50.302626 2567 scope.go:117] "RemoveContainer" containerID="79ee1af26893305254fac413f967d0b1c14740c99c82ae74994cd1e148e265b3" Sep 18 06:54:50.302985 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:54:50.302964 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79ee1af26893305254fac413f967d0b1c14740c99c82ae74994cd1e148e265b3\": container with ID starting with 79ee1af26893305254fac413f967d0b1c14740c99c82ae74994cd1e148e265b3 not found: ID does not exist" containerID="79ee1af26893305254fac413f967d0b1c14740c99c82ae74994cd1e148e265b3" Sep 18 06:54:50.303055 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:50.302997 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79ee1af26893305254fac413f967d0b1c14740c99c82ae74994cd1e148e265b3"} err="failed to get container status \"79ee1af26893305254fac413f967d0b1c14740c99c82ae74994cd1e148e265b3\": rpc error: code = NotFound desc = could not find container \"79ee1af26893305254fac413f967d0b1c14740c99c82ae74994cd1e148e265b3\": container with ID starting with 79ee1af26893305254fac413f967d0b1c14740c99c82ae74994cd1e148e265b3 not found: ID does not exist" Sep 18 06:54:50.433798 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:50.433760 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v5wrq\" (UniqueName: \"kubernetes.io/projected/885403b9-b7b0-4cc0-aa50-9486ba22a99d-kube-api-access-v5wrq\") pod \"885403b9-b7b0-4cc0-aa50-9486ba22a99d\" (UID: \"885403b9-b7b0-4cc0-aa50-9486ba22a99d\") " Sep 18 06:54:50.433985 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:50.433856 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/885403b9-b7b0-4cc0-aa50-9486ba22a99d-plugins-config-volume\") pod \"885403b9-b7b0-4cc0-aa50-9486ba22a99d\" (UID: \"885403b9-b7b0-4cc0-aa50-9486ba22a99d\") " Sep 18 06:54:50.434244 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:50.434220 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/885403b9-b7b0-4cc0-aa50-9486ba22a99d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "885403b9-b7b0-4cc0-aa50-9486ba22a99d" (UID: "885403b9-b7b0-4cc0-aa50-9486ba22a99d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 06:54:50.435806 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:50.435787 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/885403b9-b7b0-4cc0-aa50-9486ba22a99d-kube-api-access-v5wrq" (OuterVolumeSpecName: "kube-api-access-v5wrq") pod "885403b9-b7b0-4cc0-aa50-9486ba22a99d" (UID: "885403b9-b7b0-4cc0-aa50-9486ba22a99d"). InnerVolumeSpecName "kube-api-access-v5wrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 06:54:50.447616 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:50.447593 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1dadf7d7-cd98-4e5b-b478-98546ac71a29" path="/var/lib/kubelet/pods/1dadf7d7-cd98-4e5b-b478-98546ac71a29/volumes" Sep 18 06:54:50.535414 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:50.535331 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5wrq\" (UniqueName: \"kubernetes.io/projected/885403b9-b7b0-4cc0-aa50-9486ba22a99d-kube-api-access-v5wrq\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:54:50.535414 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:50.535356 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/885403b9-b7b0-4cc0-aa50-9486ba22a99d-plugins-config-volume\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:54:50.603659 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:50.603633 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-654463-db8fb8547-x6klj"] Sep 18 06:54:50.606046 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:50.606024 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-654463-db8fb8547-x6klj"] Sep 18 06:54:51.441404 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:51.441384 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-87d618-vbgxd" Sep 18 06:54:51.545056 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:51.545030 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a75c28e0-bbea-4757-95f6-13639e62aba4-maas-api-service-ca\") pod \"a75c28e0-bbea-4757-95f6-13639e62aba4\" (UID: \"a75c28e0-bbea-4757-95f6-13639e62aba4\") " Sep 18 06:54:51.545415 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:51.545391 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a75c28e0-bbea-4757-95f6-13639e62aba4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a75c28e0-bbea-4757-95f6-13639e62aba4" (UID: "a75c28e0-bbea-4757-95f6-13639e62aba4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 06:54:51.653238 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:51.646598 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a75c28e0-bbea-4757-95f6-13639e62aba4-maas-api-service-ca\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:54:52.306273 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:52.306246 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-87d618-vbgxd" Sep 18 06:54:52.306273 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:52.306259 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-87d618-vbgxd" event={"ID":"a75c28e0-bbea-4757-95f6-13639e62aba4","Type":"ContainerDied","Data":"cb7d9b3a8187574c739577a5336a6b860ce571f297ce2112f243e35cdbfbcbe1"} Sep 18 06:54:52.306508 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:52.306341 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb7d9b3a8187574c739577a5336a6b860ce571f297ce2112f243e35cdbfbcbe1" Sep 18 06:54:52.448492 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:52.448451 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="885403b9-b7b0-4cc0-aa50-9486ba22a99d" path="/var/lib/kubelet/pods/885403b9-b7b0-4cc0-aa50-9486ba22a99d/volumes" Sep 18 06:54:53.299649 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:53.299620 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp" Sep 18 06:54:53.318212 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:53.317914 2567 generic.go:356] "Generic (PLEG): container finished" podID="1d74f3c3-8297-4e77-9223-5bba5a963fea" containerID="593f1f924f7fc815fc1122f6c46f87bc05280aed4f7c591ec72fcdbec86a7b26" exitCode=0 Sep 18 06:54:53.318212 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:53.318054 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp" Sep 18 06:54:53.318212 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:53.318059 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp" event={"ID":"1d74f3c3-8297-4e77-9223-5bba5a963fea","Type":"ContainerDied","Data":"593f1f924f7fc815fc1122f6c46f87bc05280aed4f7c591ec72fcdbec86a7b26"} Sep 18 06:54:53.318501 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:53.318205 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp" event={"ID":"1d74f3c3-8297-4e77-9223-5bba5a963fea","Type":"ContainerDied","Data":"f06bc7f73820427928b92b3ac7c6f8a155f3a9889bcadfc977ed05556837c2b6"} Sep 18 06:54:53.318501 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:53.318306 2567 scope.go:117] "RemoveContainer" containerID="593f1f924f7fc815fc1122f6c46f87bc05280aed4f7c591ec72fcdbec86a7b26" Sep 18 06:54:53.335757 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:53.335733 2567 scope.go:117] "RemoveContainer" containerID="8f587b2dd77700a58ee42e3b2eb5fa4aac0e1280b2795c8519cffb78ff5e89cc" Sep 18 06:54:53.348990 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:53.348967 2567 scope.go:117] "RemoveContainer" containerID="593f1f924f7fc815fc1122f6c46f87bc05280aed4f7c591ec72fcdbec86a7b26" Sep 18 06:54:53.349351 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:54:53.349329 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"593f1f924f7fc815fc1122f6c46f87bc05280aed4f7c591ec72fcdbec86a7b26\": container with ID starting with 593f1f924f7fc815fc1122f6c46f87bc05280aed4f7c591ec72fcdbec86a7b26 not found: ID does not exist" containerID="593f1f924f7fc815fc1122f6c46f87bc05280aed4f7c591ec72fcdbec86a7b26" Sep 18 06:54:53.349471 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:53.349382 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"593f1f924f7fc815fc1122f6c46f87bc05280aed4f7c591ec72fcdbec86a7b26"} err="failed to get container status \"593f1f924f7fc815fc1122f6c46f87bc05280aed4f7c591ec72fcdbec86a7b26\": rpc error: code = NotFound desc = could not find container \"593f1f924f7fc815fc1122f6c46f87bc05280aed4f7c591ec72fcdbec86a7b26\": container with ID starting with 593f1f924f7fc815fc1122f6c46f87bc05280aed4f7c591ec72fcdbec86a7b26 not found: ID does not exist" Sep 18 06:54:53.349471 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:53.349401 2567 scope.go:117] "RemoveContainer" containerID="8f587b2dd77700a58ee42e3b2eb5fa4aac0e1280b2795c8519cffb78ff5e89cc" Sep 18 06:54:53.349760 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:54:53.349741 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f587b2dd77700a58ee42e3b2eb5fa4aac0e1280b2795c8519cffb78ff5e89cc\": container with ID starting with 8f587b2dd77700a58ee42e3b2eb5fa4aac0e1280b2795c8519cffb78ff5e89cc not found: ID does not exist" containerID="8f587b2dd77700a58ee42e3b2eb5fa4aac0e1280b2795c8519cffb78ff5e89cc" Sep 18 06:54:53.349837 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:53.349768 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f587b2dd77700a58ee42e3b2eb5fa4aac0e1280b2795c8519cffb78ff5e89cc"} err="failed to get container status \"8f587b2dd77700a58ee42e3b2eb5fa4aac0e1280b2795c8519cffb78ff5e89cc\": rpc error: code = NotFound desc = could not find container \"8f587b2dd77700a58ee42e3b2eb5fa4aac0e1280b2795c8519cffb78ff5e89cc\": container with ID starting with 8f587b2dd77700a58ee42e3b2eb5fa4aac0e1280b2795c8519cffb78ff5e89cc not found: ID does not exist" Sep 18 06:54:53.463649 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:53.463623 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d74f3c3-8297-4e77-9223-5bba5a963fea-istio-envoy\") pod \"1d74f3c3-8297-4e77-9223-5bba5a963fea\" (UID: \"1d74f3c3-8297-4e77-9223-5bba5a963fea\") " Sep 18 06:54:53.464045 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:53.463664 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d74f3c3-8297-4e77-9223-5bba5a963fea-istiod-ca-cert\") pod \"1d74f3c3-8297-4e77-9223-5bba5a963fea\" (UID: \"1d74f3c3-8297-4e77-9223-5bba5a963fea\") " Sep 18 06:54:53.464045 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:53.463684 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d74f3c3-8297-4e77-9223-5bba5a963fea-istio-podinfo\") pod \"1d74f3c3-8297-4e77-9223-5bba5a963fea\" (UID: \"1d74f3c3-8297-4e77-9223-5bba5a963fea\") " Sep 18 06:54:53.464045 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:53.463732 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2mg4\" (UniqueName: \"kubernetes.io/projected/1d74f3c3-8297-4e77-9223-5bba5a963fea-kube-api-access-n2mg4\") pod \"1d74f3c3-8297-4e77-9223-5bba5a963fea\" (UID: \"1d74f3c3-8297-4e77-9223-5bba5a963fea\") " Sep 18 06:54:53.464045 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:53.463754 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d74f3c3-8297-4e77-9223-5bba5a963fea-workload-socket\") pod \"1d74f3c3-8297-4e77-9223-5bba5a963fea\" (UID: \"1d74f3c3-8297-4e77-9223-5bba5a963fea\") " Sep 18 06:54:53.464045 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:53.463790 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d74f3c3-8297-4e77-9223-5bba5a963fea-istio-token\") pod \"1d74f3c3-8297-4e77-9223-5bba5a963fea\" (UID: \"1d74f3c3-8297-4e77-9223-5bba5a963fea\") " Sep 18 06:54:53.464045 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:53.463814 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d74f3c3-8297-4e77-9223-5bba5a963fea-istio-data\") pod \"1d74f3c3-8297-4e77-9223-5bba5a963fea\" (UID: \"1d74f3c3-8297-4e77-9223-5bba5a963fea\") " Sep 18 06:54:53.464045 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:53.464020 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d74f3c3-8297-4e77-9223-5bba5a963fea-credential-socket\") pod \"1d74f3c3-8297-4e77-9223-5bba5a963fea\" (UID: \"1d74f3c3-8297-4e77-9223-5bba5a963fea\") " Sep 18 06:54:53.464420 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:53.464061 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d74f3c3-8297-4e77-9223-5bba5a963fea-istio-data" (OuterVolumeSpecName: "istio-data") pod "1d74f3c3-8297-4e77-9223-5bba5a963fea" (UID: "1d74f3c3-8297-4e77-9223-5bba5a963fea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:54:53.464420 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:53.464097 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d74f3c3-8297-4e77-9223-5bba5a963fea-workload-certs\") pod \"1d74f3c3-8297-4e77-9223-5bba5a963fea\" (UID: \"1d74f3c3-8297-4e77-9223-5bba5a963fea\") " Sep 18 06:54:53.464420 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:53.464120 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d74f3c3-8297-4e77-9223-5bba5a963fea-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1d74f3c3-8297-4e77-9223-5bba5a963fea" (UID: "1d74f3c3-8297-4e77-9223-5bba5a963fea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:54:53.464420 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:53.464142 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d74f3c3-8297-4e77-9223-5bba5a963fea-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1d74f3c3-8297-4e77-9223-5bba5a963fea" (UID: "1d74f3c3-8297-4e77-9223-5bba5a963fea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 06:54:53.464420 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:53.464342 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d74f3c3-8297-4e77-9223-5bba5a963fea-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1d74f3c3-8297-4e77-9223-5bba5a963fea" (UID: "1d74f3c3-8297-4e77-9223-5bba5a963fea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:54:53.464652 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:53.464469 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d74f3c3-8297-4e77-9223-5bba5a963fea-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1d74f3c3-8297-4e77-9223-5bba5a963fea" (UID: "1d74f3c3-8297-4e77-9223-5bba5a963fea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:54:53.464652 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:53.464569 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d74f3c3-8297-4e77-9223-5bba5a963fea-workload-socket\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:54:53.464652 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:53.464587 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d74f3c3-8297-4e77-9223-5bba5a963fea-istio-data\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:54:53.464652 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:53.464602 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d74f3c3-8297-4e77-9223-5bba5a963fea-credential-socket\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:54:53.464652 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:53.464615 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d74f3c3-8297-4e77-9223-5bba5a963fea-workload-certs\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:54:53.464652 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:53.464629 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d74f3c3-8297-4e77-9223-5bba5a963fea-istiod-ca-cert\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:54:53.465943 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:53.465915 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d74f3c3-8297-4e77-9223-5bba5a963fea-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1d74f3c3-8297-4e77-9223-5bba5a963fea" (UID: "1d74f3c3-8297-4e77-9223-5bba5a963fea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:54:53.466045 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:53.465998 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d74f3c3-8297-4e77-9223-5bba5a963fea-istio-token" (OuterVolumeSpecName: "istio-token") pod "1d74f3c3-8297-4e77-9223-5bba5a963fea" (UID: "1d74f3c3-8297-4e77-9223-5bba5a963fea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 06:54:53.466045 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:53.466030 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d74f3c3-8297-4e77-9223-5bba5a963fea-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1d74f3c3-8297-4e77-9223-5bba5a963fea" (UID: "1d74f3c3-8297-4e77-9223-5bba5a963fea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 06:54:53.466426 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:53.466407 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d74f3c3-8297-4e77-9223-5bba5a963fea-kube-api-access-n2mg4" (OuterVolumeSpecName: "kube-api-access-n2mg4") pod "1d74f3c3-8297-4e77-9223-5bba5a963fea" (UID: "1d74f3c3-8297-4e77-9223-5bba5a963fea"). InnerVolumeSpecName "kube-api-access-n2mg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 06:54:53.565843 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:53.565819 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d74f3c3-8297-4e77-9223-5bba5a963fea-istio-token\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:54:53.565843 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:53.565842 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d74f3c3-8297-4e77-9223-5bba5a963fea-istio-envoy\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:54:53.565979 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:53.565851 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d74f3c3-8297-4e77-9223-5bba5a963fea-istio-podinfo\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:54:53.565979 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:53.565859 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2mg4\" (UniqueName: \"kubernetes.io/projected/1d74f3c3-8297-4e77-9223-5bba5a963fea-kube-api-access-n2mg4\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:54:53.641101 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:53.641071 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp"] Sep 18 06:54:53.643808 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:53.643786 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-bac224-openshift-default-7f8896f4bb-bcsqp"] Sep 18 06:54:54.445928 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:54.445895 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d74f3c3-8297-4e77-9223-5bba5a963fea" path="/var/lib/kubelet/pods/1d74f3c3-8297-4e77-9223-5bba5a963fea/volumes" Sep 18 06:54:56.595937 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:56.595902 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262"] Sep 18 06:54:56.596832 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:56.596792 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262" podUID="5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8" containerName="istio-proxy" containerID="cri-o://28c51a24335d3d5229316c14ae77f3a0ab67bb80a9ba4571e26fd9646dfe60ab" gracePeriod=30 Sep 18 06:54:59.255752 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:59.255721 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-87d618-5d94f66f4c-kwt2c"] Sep 18 06:54:59.256171 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:59.255931 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-87d618-5d94f66f4c-kwt2c" podUID="584f37b8-a704-4ba3-bba0-1035c9365bda" containerName="payload-processing" containerID="cri-o://c6813fb3b825e1b2bed599082a59983c8903a5331bee99e4b6ebf1ea6f75f6f1" gracePeriod=30 Sep 18 06:54:59.262142 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:59.262113 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-87d618-6c8d8db6bb-m75bv"] Sep 18 06:54:59.262483 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:59.262428 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-87d618-6c8d8db6bb-m75bv" podUID="b318b9a5-4a0a-4b3b-aa75-bc7879cebd45" containerName="payload-pre-processing" containerID="cri-o://e3e3f0eacb4d2a71b9a28e6533699d56cc7eb51791dcb650fc11f786c27620c7" gracePeriod=30 Sep 18 06:54:59.413560 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:59.413521 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-87d618-vbgxd"] Sep 18 06:54:59.417417 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:59.417381 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-87d618-vbgxd"] Sep 18 06:54:59.558012 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:59.557992 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-87d618-5d94f66f4c-kwt2c" Sep 18 06:54:59.560854 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:59.560835 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-87d618-6c8d8db6bb-m75bv" Sep 18 06:54:59.622704 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:59.622679 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/584f37b8-a704-4ba3-bba0-1035c9365bda-plugins-config-volume\") pod \"584f37b8-a704-4ba3-bba0-1035c9365bda\" (UID: \"584f37b8-a704-4ba3-bba0-1035c9365bda\") " Sep 18 06:54:59.622839 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:59.622780 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ddzbn\" (UniqueName: \"kubernetes.io/projected/584f37b8-a704-4ba3-bba0-1035c9365bda-kube-api-access-ddzbn\") pod \"584f37b8-a704-4ba3-bba0-1035c9365bda\" (UID: \"584f37b8-a704-4ba3-bba0-1035c9365bda\") " Sep 18 06:54:59.622839 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:59.622803 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j2z2w\" (UniqueName: \"kubernetes.io/projected/b318b9a5-4a0a-4b3b-aa75-bc7879cebd45-kube-api-access-j2z2w\") pod \"b318b9a5-4a0a-4b3b-aa75-bc7879cebd45\" (UID: \"b318b9a5-4a0a-4b3b-aa75-bc7879cebd45\") " Sep 18 06:54:59.622839 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:59.622826 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b318b9a5-4a0a-4b3b-aa75-bc7879cebd45-plugins-config-volume\") pod \"b318b9a5-4a0a-4b3b-aa75-bc7879cebd45\" (UID: \"b318b9a5-4a0a-4b3b-aa75-bc7879cebd45\") " Sep 18 06:54:59.623041 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:59.623017 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/584f37b8-a704-4ba3-bba0-1035c9365bda-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "584f37b8-a704-4ba3-bba0-1035c9365bda" (UID: "584f37b8-a704-4ba3-bba0-1035c9365bda"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 06:54:59.623255 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:59.623234 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b318b9a5-4a0a-4b3b-aa75-bc7879cebd45-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b318b9a5-4a0a-4b3b-aa75-bc7879cebd45" (UID: "b318b9a5-4a0a-4b3b-aa75-bc7879cebd45"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 06:54:59.624931 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:59.624911 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b318b9a5-4a0a-4b3b-aa75-bc7879cebd45-kube-api-access-j2z2w" (OuterVolumeSpecName: "kube-api-access-j2z2w") pod "b318b9a5-4a0a-4b3b-aa75-bc7879cebd45" (UID: "b318b9a5-4a0a-4b3b-aa75-bc7879cebd45"). InnerVolumeSpecName "kube-api-access-j2z2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 06:54:59.625002 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:59.624925 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/584f37b8-a704-4ba3-bba0-1035c9365bda-kube-api-access-ddzbn" (OuterVolumeSpecName: "kube-api-access-ddzbn") pod "584f37b8-a704-4ba3-bba0-1035c9365bda" (UID: "584f37b8-a704-4ba3-bba0-1035c9365bda"). InnerVolumeSpecName "kube-api-access-ddzbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 06:54:59.723587 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:59.723562 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddzbn\" (UniqueName: \"kubernetes.io/projected/584f37b8-a704-4ba3-bba0-1035c9365bda-kube-api-access-ddzbn\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:54:59.723587 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:59.723584 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2z2w\" (UniqueName: \"kubernetes.io/projected/b318b9a5-4a0a-4b3b-aa75-bc7879cebd45-kube-api-access-j2z2w\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:54:59.723728 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:59.723593 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b318b9a5-4a0a-4b3b-aa75-bc7879cebd45-plugins-config-volume\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:54:59.723728 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:54:59.723602 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/584f37b8-a704-4ba3-bba0-1035c9365bda-plugins-config-volume\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:55:00.366428 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:55:00.366396 2567 generic.go:356] "Generic (PLEG): container finished" podID="b318b9a5-4a0a-4b3b-aa75-bc7879cebd45" containerID="e3e3f0eacb4d2a71b9a28e6533699d56cc7eb51791dcb650fc11f786c27620c7" exitCode=0 Sep 18 06:55:00.366862 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:55:00.366516 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-87d618-6c8d8db6bb-m75bv" Sep 18 06:55:00.366862 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:55:00.366546 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-87d618-6c8d8db6bb-m75bv" event={"ID":"b318b9a5-4a0a-4b3b-aa75-bc7879cebd45","Type":"ContainerDied","Data":"e3e3f0eacb4d2a71b9a28e6533699d56cc7eb51791dcb650fc11f786c27620c7"} Sep 18 06:55:00.366862 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:55:00.366601 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-87d618-6c8d8db6bb-m75bv" event={"ID":"b318b9a5-4a0a-4b3b-aa75-bc7879cebd45","Type":"ContainerDied","Data":"b40d0281b031cb243624f66011fdcac6bf195a60edc79d7159c60585c2dcdf96"} Sep 18 06:55:00.366862 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:55:00.366634 2567 scope.go:117] "RemoveContainer" containerID="e3e3f0eacb4d2a71b9a28e6533699d56cc7eb51791dcb650fc11f786c27620c7" Sep 18 06:55:00.368118 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:55:00.368095 2567 generic.go:356] "Generic (PLEG): container finished" podID="584f37b8-a704-4ba3-bba0-1035c9365bda" containerID="c6813fb3b825e1b2bed599082a59983c8903a5331bee99e4b6ebf1ea6f75f6f1" exitCode=0 Sep 18 06:55:00.368210 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:55:00.368144 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-87d618-5d94f66f4c-kwt2c" event={"ID":"584f37b8-a704-4ba3-bba0-1035c9365bda","Type":"ContainerDied","Data":"c6813fb3b825e1b2bed599082a59983c8903a5331bee99e4b6ebf1ea6f75f6f1"} Sep 18 06:55:00.368210 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:55:00.368185 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-87d618-5d94f66f4c-kwt2c" event={"ID":"584f37b8-a704-4ba3-bba0-1035c9365bda","Type":"ContainerDied","Data":"3cbfdf3aef1979f1c86e16a4c49b9a6c7cac0bb3d699eabb2fbcb62ea6a3f51a"} Sep 18 06:55:00.368210 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:55:00.368146 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-87d618-5d94f66f4c-kwt2c" Sep 18 06:55:00.376545 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:55:00.376528 2567 scope.go:117] "RemoveContainer" containerID="e3e3f0eacb4d2a71b9a28e6533699d56cc7eb51791dcb650fc11f786c27620c7" Sep 18 06:55:00.376867 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:55:00.376835 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3e3f0eacb4d2a71b9a28e6533699d56cc7eb51791dcb650fc11f786c27620c7\": container with ID starting with e3e3f0eacb4d2a71b9a28e6533699d56cc7eb51791dcb650fc11f786c27620c7 not found: ID does not exist" containerID="e3e3f0eacb4d2a71b9a28e6533699d56cc7eb51791dcb650fc11f786c27620c7" Sep 18 06:55:00.376973 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:55:00.376878 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3e3f0eacb4d2a71b9a28e6533699d56cc7eb51791dcb650fc11f786c27620c7"} err="failed to get container status \"e3e3f0eacb4d2a71b9a28e6533699d56cc7eb51791dcb650fc11f786c27620c7\": rpc error: code = NotFound desc = could not find container \"e3e3f0eacb4d2a71b9a28e6533699d56cc7eb51791dcb650fc11f786c27620c7\": container with ID starting with e3e3f0eacb4d2a71b9a28e6533699d56cc7eb51791dcb650fc11f786c27620c7 not found: ID does not exist" Sep 18 06:55:00.376973 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:55:00.376903 2567 scope.go:117] "RemoveContainer" containerID="c6813fb3b825e1b2bed599082a59983c8903a5331bee99e4b6ebf1ea6f75f6f1" Sep 18 06:55:00.385800 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:55:00.385784 2567 scope.go:117] "RemoveContainer" containerID="c6813fb3b825e1b2bed599082a59983c8903a5331bee99e4b6ebf1ea6f75f6f1" Sep 18 06:55:00.386073 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:55:00.386054 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6813fb3b825e1b2bed599082a59983c8903a5331bee99e4b6ebf1ea6f75f6f1\": container with ID starting with c6813fb3b825e1b2bed599082a59983c8903a5331bee99e4b6ebf1ea6f75f6f1 not found: ID does not exist" containerID="c6813fb3b825e1b2bed599082a59983c8903a5331bee99e4b6ebf1ea6f75f6f1" Sep 18 06:55:00.386122 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:55:00.386080 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6813fb3b825e1b2bed599082a59983c8903a5331bee99e4b6ebf1ea6f75f6f1"} err="failed to get container status \"c6813fb3b825e1b2bed599082a59983c8903a5331bee99e4b6ebf1ea6f75f6f1\": rpc error: code = NotFound desc = could not find container \"c6813fb3b825e1b2bed599082a59983c8903a5331bee99e4b6ebf1ea6f75f6f1\": container with ID starting with c6813fb3b825e1b2bed599082a59983c8903a5331bee99e4b6ebf1ea6f75f6f1 not found: ID does not exist" Sep 18 06:55:00.393426 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:55:00.393359 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-87d618-6c8d8db6bb-m75bv"] Sep 18 06:55:00.395990 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:55:00.395970 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-87d618-6c8d8db6bb-m75bv"] Sep 18 06:55:00.404838 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:55:00.404818 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-87d618-5d94f66f4c-kwt2c"] Sep 18 06:55:00.408282 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:55:00.408260 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-87d618-5d94f66f4c-kwt2c"] Sep 18 06:55:00.447260 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:55:00.447239 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="584f37b8-a704-4ba3-bba0-1035c9365bda" path="/var/lib/kubelet/pods/584f37b8-a704-4ba3-bba0-1035c9365bda/volumes" Sep 18 06:55:00.447618 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:55:00.447603 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a75c28e0-bbea-4757-95f6-13639e62aba4" path="/var/lib/kubelet/pods/a75c28e0-bbea-4757-95f6-13639e62aba4/volumes" Sep 18 06:55:00.447917 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:55:00.447904 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b318b9a5-4a0a-4b3b-aa75-bc7879cebd45" path="/var/lib/kubelet/pods/b318b9a5-4a0a-4b3b-aa75-bc7879cebd45/volumes" Sep 18 06:55:01.845133 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:55:01.845108 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262" Sep 18 06:55:01.946289 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:55:01.946216 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wkzb6\" (UniqueName: \"kubernetes.io/projected/5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8-kube-api-access-wkzb6\") pod \"5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8\" (UID: \"5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8\") " Sep 18 06:55:01.946289 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:55:01.946255 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8-workload-certs\") pod \"5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8\" (UID: \"5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8\") " Sep 18 06:55:01.946289 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:55:01.946283 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8-credential-socket\") pod \"5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8\" (UID: \"5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8\") " Sep 18 06:55:01.946574 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:55:01.946321 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8-istio-podinfo\") pod \"5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8\" (UID: \"5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8\") " Sep 18 06:55:01.946574 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:55:01.946362 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8-istio-token\") pod \"5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8\" (UID: \"5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8\") " Sep 18 06:55:01.946574 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:55:01.946417 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8-istiod-ca-cert\") pod \"5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8\" (UID: \"5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8\") " Sep 18 06:55:01.946574 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:55:01.946449 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8-istio-data\") pod \"5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8\" (UID: \"5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8\") " Sep 18 06:55:01.946574 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:55:01.946493 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8" (UID: "5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:55:01.946574 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:55:01.946547 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8-workload-socket\") pod \"5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8\" (UID: \"5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8\") " Sep 18 06:55:01.946894 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:55:01.946590 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8-istio-envoy\") pod \"5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8\" (UID: \"5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8\") " Sep 18 06:55:01.946894 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:55:01.946689 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8" (UID: "5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:55:01.947023 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:55:01.946954 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8" (UID: "5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 06:55:01.947023 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:55:01.947006 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8-workload-certs\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:55:01.947736 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:55:01.947028 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8-credential-socket\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:55:01.947736 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:55:01.947188 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8" (UID: "5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:55:01.947736 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:55:01.947256 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8-istio-data" (OuterVolumeSpecName: "istio-data") pod "5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8" (UID: "5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:55:01.948645 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:55:01.948616 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8-kube-api-access-wkzb6" (OuterVolumeSpecName: "kube-api-access-wkzb6") pod "5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8" (UID: "5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8"). InnerVolumeSpecName "kube-api-access-wkzb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 06:55:01.948759 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:55:01.948672 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8" (UID: "5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 06:55:01.948815 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:55:01.948782 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8-istio-token" (OuterVolumeSpecName: "istio-token") pod "5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8" (UID: "5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 06:55:01.948926 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:55:01.948904 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8" (UID: "5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 06:55:02.048169 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:55:02.048115 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8-istiod-ca-cert\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:55:02.048169 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:55:02.048135 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8-istio-data\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:55:02.048169 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:55:02.048143 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8-workload-socket\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:55:02.048169 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:55:02.048152 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8-istio-envoy\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:55:02.048169 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:55:02.048161 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkzb6\" (UniqueName: \"kubernetes.io/projected/5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8-kube-api-access-wkzb6\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:55:02.048419 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:55:02.048171 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8-istio-podinfo\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:55:02.048419 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:55:02.048184 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8-istio-token\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 06:55:02.385521 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:55:02.385488 2567 generic.go:356] "Generic (PLEG): container finished" podID="5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8" containerID="28c51a24335d3d5229316c14ae77f3a0ab67bb80a9ba4571e26fd9646dfe60ab" exitCode=0 Sep 18 06:55:02.385713 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:55:02.385525 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262" event={"ID":"5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8","Type":"ContainerDied","Data":"28c51a24335d3d5229316c14ae77f3a0ab67bb80a9ba4571e26fd9646dfe60ab"} Sep 18 06:55:02.385713 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:55:02.385549 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262" Sep 18 06:55:02.385713 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:55:02.385609 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262" event={"ID":"5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8","Type":"ContainerDied","Data":"d8ca0bbb2c2beebc33210d6d4291c3caca32708e604c1de2264524fde9739fc9"} Sep 18 06:55:02.385713 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:55:02.385632 2567 scope.go:117] "RemoveContainer" containerID="28c51a24335d3d5229316c14ae77f3a0ab67bb80a9ba4571e26fd9646dfe60ab" Sep 18 06:55:02.397016 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:55:02.396995 2567 scope.go:117] "RemoveContainer" containerID="d45ebdf186426c3e3c1691771b762497d85f1643093e5d0f4fc83446874b6dca" Sep 18 06:55:02.406161 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:55:02.406130 2567 scope.go:117] "RemoveContainer" containerID="28c51a24335d3d5229316c14ae77f3a0ab67bb80a9ba4571e26fd9646dfe60ab" Sep 18 06:55:02.406452 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:55:02.406433 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28c51a24335d3d5229316c14ae77f3a0ab67bb80a9ba4571e26fd9646dfe60ab\": container with ID starting with 28c51a24335d3d5229316c14ae77f3a0ab67bb80a9ba4571e26fd9646dfe60ab not found: ID does not exist" containerID="28c51a24335d3d5229316c14ae77f3a0ab67bb80a9ba4571e26fd9646dfe60ab" Sep 18 06:55:02.406518 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:55:02.406461 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28c51a24335d3d5229316c14ae77f3a0ab67bb80a9ba4571e26fd9646dfe60ab"} err="failed to get container status \"28c51a24335d3d5229316c14ae77f3a0ab67bb80a9ba4571e26fd9646dfe60ab\": rpc error: code = NotFound desc = could not find container \"28c51a24335d3d5229316c14ae77f3a0ab67bb80a9ba4571e26fd9646dfe60ab\": container with ID starting with 28c51a24335d3d5229316c14ae77f3a0ab67bb80a9ba4571e26fd9646dfe60ab not found: ID does not exist" Sep 18 06:55:02.406518 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:55:02.406479 2567 scope.go:117] "RemoveContainer" containerID="d45ebdf186426c3e3c1691771b762497d85f1643093e5d0f4fc83446874b6dca" Sep 18 06:55:02.406735 ip-10-0-142-104 kubenswrapper[2567]: E0918 06:55:02.406714 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d45ebdf186426c3e3c1691771b762497d85f1643093e5d0f4fc83446874b6dca\": container with ID starting with d45ebdf186426c3e3c1691771b762497d85f1643093e5d0f4fc83446874b6dca not found: ID does not exist" containerID="d45ebdf186426c3e3c1691771b762497d85f1643093e5d0f4fc83446874b6dca" Sep 18 06:55:02.406823 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:55:02.406738 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d45ebdf186426c3e3c1691771b762497d85f1643093e5d0f4fc83446874b6dca"} err="failed to get container status \"d45ebdf186426c3e3c1691771b762497d85f1643093e5d0f4fc83446874b6dca\": rpc error: code = NotFound desc = could not find container \"d45ebdf186426c3e3c1691771b762497d85f1643093e5d0f4fc83446874b6dca\": container with ID starting with d45ebdf186426c3e3c1691771b762497d85f1643093e5d0f4fc83446874b6dca not found: ID does not exist" Sep 18 06:55:02.408222 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:55:02.408203 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262"] Sep 18 06:55:02.413299 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:55:02.413265 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-654463-openshift-default-8bf48d448-d6262"] Sep 18 06:55:02.445769 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:55:02.445748 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8" path="/var/lib/kubelet/pods/5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8/volumes" Sep 18 06:55:20.831477 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:55:20.831442 2567 scope.go:117] "RemoveContainer" containerID="244e248746694d6da7a9857870cc225dd90d7fb9545dc6fefa63204c31021aa1" Sep 18 06:56:20.909063 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:56:20.909027 2567 scope.go:117] "RemoveContainer" containerID="ba7650920d50ce0a02a394e6f6dc0b0c54ad5c3e4521014228cc9d279f2d2c0c" Sep 18 06:57:20.600204 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:57:20.600175 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjr4h_eb059951-e263-40da-9155-4fd0cf242820/ovn-acl-logging/0.log" Sep 18 06:57:20.605285 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:57:20.605261 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjr4h_eb059951-e263-40da-9155-4fd0cf242820/ovn-acl-logging/0.log" Sep 18 06:58:41.355575 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:41.355537 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk"] Sep 18 06:58:41.356143 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:41.356125 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8" containerName="istio-proxy" Sep 18 06:58:41.356216 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:41.356145 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8" containerName="istio-proxy" Sep 18 06:58:41.356216 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:41.356157 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d74f3c3-8297-4e77-9223-5bba5a963fea" containerName="istio-proxy" Sep 18 06:58:41.356216 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:41.356166 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d74f3c3-8297-4e77-9223-5bba5a963fea" containerName="istio-proxy" Sep 18 06:58:41.356216 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:41.356189 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b318b9a5-4a0a-4b3b-aa75-bc7879cebd45" containerName="payload-pre-processing" Sep 18 06:58:41.356216 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:41.356198 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b318b9a5-4a0a-4b3b-aa75-bc7879cebd45" containerName="payload-pre-processing" Sep 18 06:58:41.356216 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:41.356210 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="584f37b8-a704-4ba3-bba0-1035c9365bda" containerName="payload-processing" Sep 18 06:58:41.356464 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:41.356220 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="584f37b8-a704-4ba3-bba0-1035c9365bda" containerName="payload-processing" Sep 18 06:58:41.356464 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:41.356239 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8" containerName="istio-proxy" Sep 18 06:58:41.356464 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:41.356248 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8" containerName="istio-proxy" Sep 18 06:58:41.356464 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:41.356267 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="885403b9-b7b0-4cc0-aa50-9486ba22a99d" containerName="payload-processing" Sep 18 06:58:41.356464 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:41.356276 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="885403b9-b7b0-4cc0-aa50-9486ba22a99d" containerName="payload-processing" Sep 18 06:58:41.356464 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:41.356287 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a75c28e0-bbea-4757-95f6-13639e62aba4" containerName="revoke-keys" Sep 18 06:58:41.356464 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:41.356295 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a75c28e0-bbea-4757-95f6-13639e62aba4" containerName="revoke-keys" Sep 18 06:58:41.356464 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:41.356422 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="885403b9-b7b0-4cc0-aa50-9486ba22a99d" containerName="payload-processing" Sep 18 06:58:41.356464 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:41.356434 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d74f3c3-8297-4e77-9223-5bba5a963fea" containerName="istio-proxy" Sep 18 06:58:41.356464 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:41.356444 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d74f3c3-8297-4e77-9223-5bba5a963fea" containerName="istio-proxy" Sep 18 06:58:41.356464 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:41.356454 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8" containerName="istio-proxy" Sep 18 06:58:41.356464 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:41.356462 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="b318b9a5-4a0a-4b3b-aa75-bc7879cebd45" containerName="payload-pre-processing" Sep 18 06:58:41.356824 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:41.356471 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a8b52a3-9d9a-4a7c-bfb7-5faca89c2cc8" containerName="istio-proxy" Sep 18 06:58:41.356824 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:41.356481 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a75c28e0-bbea-4757-95f6-13639e62aba4" containerName="revoke-keys" Sep 18 06:58:41.356824 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:41.356490 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="584f37b8-a704-4ba3-bba0-1035c9365bda" containerName="payload-processing" Sep 18 06:58:41.360067 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:41.360045 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk" Sep 18 06:58:41.362656 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:41.362630 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-a7bd58-openshift-default-dockercfg-g79tw\"" Sep 18 06:58:41.367294 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:41.367267 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk"] Sep 18 06:58:41.381103 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:41.381068 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcd19467-e9c3-462b-b277-9cb88354fc88-istio-data\") pod \"e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk\" (UID: \"fcd19467-e9c3-462b-b277-9cb88354fc88\") " pod="openshift-ingress/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk" Sep 18 06:58:41.381275 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:41.381111 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcd19467-e9c3-462b-b277-9cb88354fc88-workload-socket\") pod \"e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk\" (UID: \"fcd19467-e9c3-462b-b277-9cb88354fc88\") " pod="openshift-ingress/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk" Sep 18 06:58:41.381275 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:41.381136 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcd19467-e9c3-462b-b277-9cb88354fc88-workload-certs\") pod \"e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk\" (UID: \"fcd19467-e9c3-462b-b277-9cb88354fc88\") " pod="openshift-ingress/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk" Sep 18 06:58:41.381275 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:41.381189 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szl8d\" (UniqueName: \"kubernetes.io/projected/fcd19467-e9c3-462b-b277-9cb88354fc88-kube-api-access-szl8d\") pod \"e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk\" (UID: \"fcd19467-e9c3-462b-b277-9cb88354fc88\") " pod="openshift-ingress/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk" Sep 18 06:58:41.381275 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:41.381226 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcd19467-e9c3-462b-b277-9cb88354fc88-istio-podinfo\") pod \"e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk\" (UID: \"fcd19467-e9c3-462b-b277-9cb88354fc88\") " pod="openshift-ingress/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk" Sep 18 06:58:41.381275 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:41.381254 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcd19467-e9c3-462b-b277-9cb88354fc88-credential-socket\") pod \"e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk\" (UID: \"fcd19467-e9c3-462b-b277-9cb88354fc88\") " pod="openshift-ingress/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk" Sep 18 06:58:41.381612 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:41.381333 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcd19467-e9c3-462b-b277-9cb88354fc88-istio-token\") pod \"e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk\" (UID: \"fcd19467-e9c3-462b-b277-9cb88354fc88\") " pod="openshift-ingress/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk" Sep 18 06:58:41.381612 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:41.381348 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcd19467-e9c3-462b-b277-9cb88354fc88-istiod-ca-cert\") pod \"e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk\" (UID: \"fcd19467-e9c3-462b-b277-9cb88354fc88\") " pod="openshift-ingress/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk" Sep 18 06:58:41.381612 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:41.381426 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcd19467-e9c3-462b-b277-9cb88354fc88-istio-envoy\") pod \"e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk\" (UID: \"fcd19467-e9c3-462b-b277-9cb88354fc88\") " pod="openshift-ingress/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk" Sep 18 06:58:41.482641 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:41.482602 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcd19467-e9c3-462b-b277-9cb88354fc88-istio-envoy\") pod \"e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk\" (UID: \"fcd19467-e9c3-462b-b277-9cb88354fc88\") " pod="openshift-ingress/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk" Sep 18 06:58:41.482641 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:41.482647 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcd19467-e9c3-462b-b277-9cb88354fc88-istio-data\") pod \"e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk\" (UID: \"fcd19467-e9c3-462b-b277-9cb88354fc88\") " pod="openshift-ingress/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk" Sep 18 06:58:41.482921 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:41.482667 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcd19467-e9c3-462b-b277-9cb88354fc88-workload-socket\") pod \"e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk\" (UID: \"fcd19467-e9c3-462b-b277-9cb88354fc88\") " pod="openshift-ingress/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk" Sep 18 06:58:41.482921 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:41.482683 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcd19467-e9c3-462b-b277-9cb88354fc88-workload-certs\") pod \"e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk\" (UID: \"fcd19467-e9c3-462b-b277-9cb88354fc88\") " pod="openshift-ingress/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk" Sep 18 06:58:41.482921 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:41.482741 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szl8d\" (UniqueName: \"kubernetes.io/projected/fcd19467-e9c3-462b-b277-9cb88354fc88-kube-api-access-szl8d\") pod \"e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk\" (UID: \"fcd19467-e9c3-462b-b277-9cb88354fc88\") " pod="openshift-ingress/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk" Sep 18 06:58:41.482921 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:41.482774 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcd19467-e9c3-462b-b277-9cb88354fc88-istio-podinfo\") pod \"e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk\" (UID: \"fcd19467-e9c3-462b-b277-9cb88354fc88\") " pod="openshift-ingress/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk" Sep 18 06:58:41.482921 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:41.482809 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcd19467-e9c3-462b-b277-9cb88354fc88-credential-socket\") pod \"e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk\" (UID: \"fcd19467-e9c3-462b-b277-9cb88354fc88\") " pod="openshift-ingress/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk" Sep 18 06:58:41.482921 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:41.482876 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcd19467-e9c3-462b-b277-9cb88354fc88-istio-token\") pod \"e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk\" (UID: \"fcd19467-e9c3-462b-b277-9cb88354fc88\") " pod="openshift-ingress/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk" Sep 18 06:58:41.482921 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:41.482902 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcd19467-e9c3-462b-b277-9cb88354fc88-istiod-ca-cert\") pod \"e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk\" (UID: \"fcd19467-e9c3-462b-b277-9cb88354fc88\") " pod="openshift-ingress/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk" Sep 18 06:58:41.483324 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:41.483096 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcd19467-e9c3-462b-b277-9cb88354fc88-workload-socket\") pod \"e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk\" (UID: \"fcd19467-e9c3-462b-b277-9cb88354fc88\") " pod="openshift-ingress/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk" Sep 18 06:58:41.483324 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:41.483168 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcd19467-e9c3-462b-b277-9cb88354fc88-istio-data\") pod \"e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk\" (UID: \"fcd19467-e9c3-462b-b277-9cb88354fc88\") " pod="openshift-ingress/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk" Sep 18 06:58:41.483324 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:41.483205 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcd19467-e9c3-462b-b277-9cb88354fc88-workload-certs\") pod \"e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk\" (UID: \"fcd19467-e9c3-462b-b277-9cb88354fc88\") " pod="openshift-ingress/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk" Sep 18 06:58:41.483324 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:41.483214 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcd19467-e9c3-462b-b277-9cb88354fc88-credential-socket\") pod \"e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk\" (UID: \"fcd19467-e9c3-462b-b277-9cb88354fc88\") " pod="openshift-ingress/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk" Sep 18 06:58:41.483564 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:41.483550 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcd19467-e9c3-462b-b277-9cb88354fc88-istiod-ca-cert\") pod \"e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk\" (UID: \"fcd19467-e9c3-462b-b277-9cb88354fc88\") " pod="openshift-ingress/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk" Sep 18 06:58:41.485484 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:41.485463 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcd19467-e9c3-462b-b277-9cb88354fc88-istio-envoy\") pod \"e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk\" (UID: \"fcd19467-e9c3-462b-b277-9cb88354fc88\") " pod="openshift-ingress/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk" Sep 18 06:58:41.485603 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:41.485518 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcd19467-e9c3-462b-b277-9cb88354fc88-istio-podinfo\") pod \"e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk\" (UID: \"fcd19467-e9c3-462b-b277-9cb88354fc88\") " pod="openshift-ingress/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk" Sep 18 06:58:41.490616 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:41.490592 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szl8d\" (UniqueName: \"kubernetes.io/projected/fcd19467-e9c3-462b-b277-9cb88354fc88-kube-api-access-szl8d\") pod \"e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk\" (UID: \"fcd19467-e9c3-462b-b277-9cb88354fc88\") " pod="openshift-ingress/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk" Sep 18 06:58:41.490949 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:41.490928 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcd19467-e9c3-462b-b277-9cb88354fc88-istio-token\") pod \"e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk\" (UID: \"fcd19467-e9c3-462b-b277-9cb88354fc88\") " pod="openshift-ingress/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk" Sep 18 06:58:41.673681 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:41.673588 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk" Sep 18 06:58:41.955934 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:41.955905 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk"] Sep 18 06:58:41.961803 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:58:41.961772 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcd19467_e9c3_462b_b277_9cb88354fc88.slice/crio-a265667deaa9e0f7ca6c4bb9788fc57eeddbef0d0bf425c64fae2849cd5ea3f5 WatchSource:0}: Error finding container a265667deaa9e0f7ca6c4bb9788fc57eeddbef0d0bf425c64fae2849cd5ea3f5: Status 404 returned error can't find the container with id a265667deaa9e0f7ca6c4bb9788fc57eeddbef0d0bf425c64fae2849cd5ea3f5 Sep 18 06:58:41.964476 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:41.964438 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:58:41.964576 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:41.964529 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:58:41.964646 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:41.964571 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:58:42.690001 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:42.689962 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk" event={"ID":"fcd19467-e9c3-462b-b277-9cb88354fc88","Type":"ContainerStarted","Data":"2862ac6ba4054aff8f4de8c2d7d865b4955018f9f9d2a4770cfaeaf019c8991f"} Sep 18 06:58:42.690428 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:42.690028 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk" event={"ID":"fcd19467-e9c3-462b-b277-9cb88354fc88","Type":"ContainerStarted","Data":"a265667deaa9e0f7ca6c4bb9788fc57eeddbef0d0bf425c64fae2849cd5ea3f5"} Sep 18 06:58:42.712341 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:42.712287 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk" podStartSLOduration=1.7122692910000001 podStartE2EDuration="1.712269291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:58:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:58:42.711133694 +0000 UTC m=+2184.450549896" watchObservedRunningTime="2026-09-18 06:58:42.712269291 +0000 UTC m=+2184.451685466" Sep 18 06:58:43.673971 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:43.673934 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk" Sep 18 06:58:43.679612 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:43.679584 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk" Sep 18 06:58:43.695987 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:43.695958 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk" Sep 18 06:58:43.696881 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:43.696859 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk" Sep 18 06:58:46.531144 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:46.531105 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a7bd58-54468db6f8-4tdxf"] Sep 18 06:58:46.531607 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:46.531586 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d74f3c3-8297-4e77-9223-5bba5a963fea" containerName="istio-proxy" Sep 18 06:58:46.531670 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:46.531609 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d74f3c3-8297-4e77-9223-5bba5a963fea" containerName="istio-proxy" Sep 18 06:58:46.534610 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:46.534583 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a7bd58-54468db6f8-4tdxf" Sep 18 06:58:46.537561 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:46.537530 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a7bd58\"" Sep 18 06:58:46.556455 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:46.556425 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a7bd58-54468db6f8-4tdxf"] Sep 18 06:58:46.574209 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:46.574173 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a7bd58-6999dbc775-qcqlf"] Sep 18 06:58:46.577848 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:46.577820 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a7bd58-6999dbc775-qcqlf" Sep 18 06:58:46.580891 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:46.580864 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-a7bd58-dockercfg-2ct6x\"" Sep 18 06:58:46.601973 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:46.601939 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a7bd58-6999dbc775-qcqlf"] Sep 18 06:58:46.637888 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:46.637849 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trvc4\" (UniqueName: \"kubernetes.io/projected/d78179e1-1ee6-4767-9496-8a1973cb590a-kube-api-access-trvc4\") pod \"payload-pre-processing-e2e-ipp-a-a7bd58-54468db6f8-4tdxf\" (UID: \"d78179e1-1ee6-4767-9496-8a1973cb590a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a7bd58-54468db6f8-4tdxf" Sep 18 06:58:46.637888 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:46.637884 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frc2c\" (UniqueName: \"kubernetes.io/projected/98d614d7-ff85-4f64-b4a2-d85549dd05af-kube-api-access-frc2c\") pod \"payload-processing-e2e-ipp-a-a7bd58-6999dbc775-qcqlf\" (UID: \"98d614d7-ff85-4f64-b4a2-d85549dd05af\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a7bd58-6999dbc775-qcqlf" Sep 18 06:58:46.638115 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:46.637999 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d78179e1-1ee6-4767-9496-8a1973cb590a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a7bd58-54468db6f8-4tdxf\" (UID: \"d78179e1-1ee6-4767-9496-8a1973cb590a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a7bd58-54468db6f8-4tdxf" Sep 18 06:58:46.638115 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:46.638047 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98d614d7-ff85-4f64-b4a2-d85549dd05af-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a7bd58-6999dbc775-qcqlf\" (UID: \"98d614d7-ff85-4f64-b4a2-d85549dd05af\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a7bd58-6999dbc775-qcqlf" Sep 18 06:58:46.739835 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:46.739792 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trvc4\" (UniqueName: \"kubernetes.io/projected/d78179e1-1ee6-4767-9496-8a1973cb590a-kube-api-access-trvc4\") pod \"payload-pre-processing-e2e-ipp-a-a7bd58-54468db6f8-4tdxf\" (UID: \"d78179e1-1ee6-4767-9496-8a1973cb590a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a7bd58-54468db6f8-4tdxf" Sep 18 06:58:46.739835 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:46.739838 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frc2c\" (UniqueName: \"kubernetes.io/projected/98d614d7-ff85-4f64-b4a2-d85549dd05af-kube-api-access-frc2c\") pod \"payload-processing-e2e-ipp-a-a7bd58-6999dbc775-qcqlf\" (UID: \"98d614d7-ff85-4f64-b4a2-d85549dd05af\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a7bd58-6999dbc775-qcqlf" Sep 18 06:58:46.740075 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:46.739878 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d78179e1-1ee6-4767-9496-8a1973cb590a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a7bd58-54468db6f8-4tdxf\" (UID: \"d78179e1-1ee6-4767-9496-8a1973cb590a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a7bd58-54468db6f8-4tdxf" Sep 18 06:58:46.740075 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:46.739900 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98d614d7-ff85-4f64-b4a2-d85549dd05af-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a7bd58-6999dbc775-qcqlf\" (UID: \"98d614d7-ff85-4f64-b4a2-d85549dd05af\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a7bd58-6999dbc775-qcqlf" Sep 18 06:58:46.740681 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:46.740662 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98d614d7-ff85-4f64-b4a2-d85549dd05af-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a7bd58-6999dbc775-qcqlf\" (UID: \"98d614d7-ff85-4f64-b4a2-d85549dd05af\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a7bd58-6999dbc775-qcqlf" Sep 18 06:58:46.740726 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:46.740668 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d78179e1-1ee6-4767-9496-8a1973cb590a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a7bd58-54468db6f8-4tdxf\" (UID: \"d78179e1-1ee6-4767-9496-8a1973cb590a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a7bd58-54468db6f8-4tdxf" Sep 18 06:58:46.747325 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:46.747292 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trvc4\" (UniqueName: \"kubernetes.io/projected/d78179e1-1ee6-4767-9496-8a1973cb590a-kube-api-access-trvc4\") pod \"payload-pre-processing-e2e-ipp-a-a7bd58-54468db6f8-4tdxf\" (UID: \"d78179e1-1ee6-4767-9496-8a1973cb590a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a7bd58-54468db6f8-4tdxf" Sep 18 06:58:46.747495 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:46.747356 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frc2c\" (UniqueName: \"kubernetes.io/projected/98d614d7-ff85-4f64-b4a2-d85549dd05af-kube-api-access-frc2c\") pod \"payload-processing-e2e-ipp-a-a7bd58-6999dbc775-qcqlf\" (UID: \"98d614d7-ff85-4f64-b4a2-d85549dd05af\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a7bd58-6999dbc775-qcqlf" Sep 18 06:58:46.845334 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:46.845295 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a7bd58-54468db6f8-4tdxf" Sep 18 06:58:46.890692 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:46.890653 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a7bd58-6999dbc775-qcqlf" Sep 18 06:58:47.125648 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:47.125616 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a7bd58-54468db6f8-4tdxf"] Sep 18 06:58:47.127881 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:58:47.127851 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd78179e1_1ee6_4767_9496_8a1973cb590a.slice/crio-039174019fd8757586e1f98f5ed96589a05069e031a9f5638501b8c10dc9d5cc WatchSource:0}: Error finding container 039174019fd8757586e1f98f5ed96589a05069e031a9f5638501b8c10dc9d5cc: Status 404 returned error can't find the container with id 039174019fd8757586e1f98f5ed96589a05069e031a9f5638501b8c10dc9d5cc Sep 18 06:58:47.180563 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:47.180537 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a7bd58-6999dbc775-qcqlf"] Sep 18 06:58:47.182228 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:58:47.182204 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98d614d7_ff85_4f64_b4a2_d85549dd05af.slice/crio-65fb443cc10aa723df6965f0ab88b238946a59b708bfd8a16b8531c8b1096787 WatchSource:0}: Error finding container 65fb443cc10aa723df6965f0ab88b238946a59b708bfd8a16b8531c8b1096787: Status 404 returned error can't find the container with id 65fb443cc10aa723df6965f0ab88b238946a59b708bfd8a16b8531c8b1096787 Sep 18 06:58:47.721035 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:47.720983 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a7bd58-6999dbc775-qcqlf" event={"ID":"98d614d7-ff85-4f64-b4a2-d85549dd05af","Type":"ContainerStarted","Data":"f39934de0c56519be35d256757e854d61035356e9577120ffa02e03e356f117d"} Sep 18 06:58:47.721532 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:47.721052 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a7bd58-6999dbc775-qcqlf" event={"ID":"98d614d7-ff85-4f64-b4a2-d85549dd05af","Type":"ContainerStarted","Data":"65fb443cc10aa723df6965f0ab88b238946a59b708bfd8a16b8531c8b1096787"} Sep 18 06:58:47.721532 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:47.721089 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a7bd58-6999dbc775-qcqlf" Sep 18 06:58:47.722862 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:47.722836 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a7bd58-54468db6f8-4tdxf" event={"ID":"d78179e1-1ee6-4767-9496-8a1973cb590a","Type":"ContainerStarted","Data":"43670c11305976d4c1019639d32256419f4ff9ef83ee82cb128482c96a048f6c"} Sep 18 06:58:47.722978 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:47.722877 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a7bd58-54468db6f8-4tdxf" event={"ID":"d78179e1-1ee6-4767-9496-8a1973cb590a","Type":"ContainerStarted","Data":"039174019fd8757586e1f98f5ed96589a05069e031a9f5638501b8c10dc9d5cc"} Sep 18 06:58:47.722978 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:47.722897 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a7bd58-54468db6f8-4tdxf" Sep 18 06:58:47.740419 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:47.740345 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-a7bd58-6999dbc775-qcqlf" podStartSLOduration=1.740324829 podStartE2EDuration="1.740324829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:58:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:58:47.737425724 +0000 UTC m=+2189.476841921" watchObservedRunningTime="2026-09-18 06:58:47.740324829 +0000 UTC m=+2189.479741007" Sep 18 06:58:47.752141 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:47.752090 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a7bd58-54468db6f8-4tdxf" podStartSLOduration=1.75207295 podStartE2EDuration="1.75207295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:58:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:58:47.751726693 +0000 UTC m=+2189.491142896" watchObservedRunningTime="2026-09-18 06:58:47.75207295 +0000 UTC m=+2189.491489134" Sep 18 06:58:58.731653 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:58.731610 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a7bd58-6999dbc775-qcqlf" Sep 18 06:58:58.732103 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:58.731701 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a7bd58-54468db6f8-4tdxf" Sep 18 06:58:59.737621 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:59.737568 2567 patch_prober.go:28] interesting pod/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 18 06:58:59.738978 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:58:59.737651 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk" podUID="fcd19467-e9c3-462b-b277-9cb88354fc88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 18 06:59:00.817984 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:59:00.817948 2567 generic.go:356] "Generic (PLEG): container finished" podID="fcd19467-e9c3-462b-b277-9cb88354fc88" containerID="2862ac6ba4054aff8f4de8c2d7d865b4955018f9f9d2a4770cfaeaf019c8991f" exitCode=0 Sep 18 06:59:00.818454 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:59:00.818022 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk" event={"ID":"fcd19467-e9c3-462b-b277-9cb88354fc88","Type":"ContainerDied","Data":"2862ac6ba4054aff8f4de8c2d7d865b4955018f9f9d2a4770cfaeaf019c8991f"} Sep 18 06:59:00.818543 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:59:00.818474 2567 scope.go:117] "RemoveContainer" containerID="2862ac6ba4054aff8f4de8c2d7d865b4955018f9f9d2a4770cfaeaf019c8991f" Sep 18 06:59:00.819586 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:59:00.819557 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:59:00.819681 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:59:00.819621 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:59:00.819681 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:59:00.819652 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 06:59:01.826006 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:59:01.825968 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk" event={"ID":"fcd19467-e9c3-462b-b277-9cb88354fc88","Type":"ContainerStarted","Data":"3df974ca684cdbaa6aa368e682417fa04f19c98a91c9170477800d85b68d1e6d"} Sep 18 06:59:02.674008 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:59:02.673960 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk" Sep 18 06:59:02.674227 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:59:02.674202 2567 patch_prober.go:28] interesting pod/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Sep 18 06:59:02.674277 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:59:02.674257 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk" podUID="fcd19467-e9c3-462b-b277-9cb88354fc88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Sep 18 06:59:03.674265 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:59:03.674225 2567 patch_prober.go:28] interesting pod/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Sep 18 06:59:03.674704 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:59:03.674286 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk" podUID="fcd19467-e9c3-462b-b277-9cb88354fc88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Sep 18 06:59:04.674293 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:59:04.674257 2567 patch_prober.go:28] interesting pod/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Sep 18 06:59:04.674708 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:59:04.674314 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk" podUID="fcd19467-e9c3-462b-b277-9cb88354fc88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Sep 18 06:59:05.674294 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:59:05.674246 2567 patch_prober.go:28] interesting pod/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Sep 18 06:59:05.674505 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:59:05.674320 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk" podUID="fcd19467-e9c3-462b-b277-9cb88354fc88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Sep 18 06:59:06.674250 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:59:06.674214 2567 patch_prober.go:28] interesting pod/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Sep 18 06:59:06.674454 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:59:06.674273 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk" podUID="fcd19467-e9c3-462b-b277-9cb88354fc88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Sep 18 06:59:07.677758 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:59:07.677717 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk" Sep 18 06:59:07.678191 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:59:07.678171 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk" Sep 18 06:59:07.678981 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:59:07.678957 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk" Sep 18 06:59:16.461207 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:59:16.461165 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1672e6-b445c6b9d-lsp7s"] Sep 18 06:59:16.463703 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:59:16.463686 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1672e6-b445c6b9d-lsp7s" Sep 18 06:59:16.465979 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:59:16.465958 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-1672e6\"" Sep 18 06:59:16.476028 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:59:16.476002 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1672e6-b445c6b9d-lsp7s"] Sep 18 06:59:16.514781 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:59:16.514704 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1672e6-58bfdf58c7-7mvlb"] Sep 18 06:59:16.517898 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:59:16.517870 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1672e6-58bfdf58c7-7mvlb" Sep 18 06:59:16.520224 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:59:16.520178 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-1672e6-dockercfg-86vdw\"" Sep 18 06:59:16.528118 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:59:16.528091 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1672e6-58bfdf58c7-7mvlb"] Sep 18 06:59:16.536471 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:59:16.536435 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45abc740-d22c-4fcc-af21-e75a01952cd6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1672e6-b445c6b9d-lsp7s\" (UID: \"45abc740-d22c-4fcc-af21-e75a01952cd6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1672e6-b445c6b9d-lsp7s" Sep 18 06:59:16.536658 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:59:16.536552 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzn6z\" (UniqueName: \"kubernetes.io/projected/45abc740-d22c-4fcc-af21-e75a01952cd6-kube-api-access-lzn6z\") pod \"payload-pre-processing-e2e-ipp-b-1672e6-b445c6b9d-lsp7s\" (UID: \"45abc740-d22c-4fcc-af21-e75a01952cd6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1672e6-b445c6b9d-lsp7s" Sep 18 06:59:16.536658 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:59:16.536609 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptjd6\" (UniqueName: \"kubernetes.io/projected/c1e71afc-e8f0-4bc8-b1b1-93d16f5c1641-kube-api-access-ptjd6\") pod \"payload-processing-e2e-ipp-b-1672e6-58bfdf58c7-7mvlb\" (UID: \"c1e71afc-e8f0-4bc8-b1b1-93d16f5c1641\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1672e6-58bfdf58c7-7mvlb" Sep 18 06:59:16.536764 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:59:16.536695 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1e71afc-e8f0-4bc8-b1b1-93d16f5c1641-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1672e6-58bfdf58c7-7mvlb\" (UID: \"c1e71afc-e8f0-4bc8-b1b1-93d16f5c1641\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1672e6-58bfdf58c7-7mvlb" Sep 18 06:59:16.637982 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:59:16.637943 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzn6z\" (UniqueName: \"kubernetes.io/projected/45abc740-d22c-4fcc-af21-e75a01952cd6-kube-api-access-lzn6z\") pod \"payload-pre-processing-e2e-ipp-b-1672e6-b445c6b9d-lsp7s\" (UID: \"45abc740-d22c-4fcc-af21-e75a01952cd6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1672e6-b445c6b9d-lsp7s" Sep 18 06:59:16.638167 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:59:16.638004 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptjd6\" (UniqueName: \"kubernetes.io/projected/c1e71afc-e8f0-4bc8-b1b1-93d16f5c1641-kube-api-access-ptjd6\") pod \"payload-processing-e2e-ipp-b-1672e6-58bfdf58c7-7mvlb\" (UID: \"c1e71afc-e8f0-4bc8-b1b1-93d16f5c1641\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1672e6-58bfdf58c7-7mvlb" Sep 18 06:59:16.638167 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:59:16.638049 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1e71afc-e8f0-4bc8-b1b1-93d16f5c1641-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1672e6-58bfdf58c7-7mvlb\" (UID: \"c1e71afc-e8f0-4bc8-b1b1-93d16f5c1641\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1672e6-58bfdf58c7-7mvlb" Sep 18 06:59:16.638167 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:59:16.638112 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45abc740-d22c-4fcc-af21-e75a01952cd6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1672e6-b445c6b9d-lsp7s\" (UID: \"45abc740-d22c-4fcc-af21-e75a01952cd6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1672e6-b445c6b9d-lsp7s" Sep 18 06:59:16.638880 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:59:16.638860 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1e71afc-e8f0-4bc8-b1b1-93d16f5c1641-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1672e6-58bfdf58c7-7mvlb\" (UID: \"c1e71afc-e8f0-4bc8-b1b1-93d16f5c1641\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1672e6-58bfdf58c7-7mvlb" Sep 18 06:59:16.638938 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:59:16.638866 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45abc740-d22c-4fcc-af21-e75a01952cd6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1672e6-b445c6b9d-lsp7s\" (UID: \"45abc740-d22c-4fcc-af21-e75a01952cd6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1672e6-b445c6b9d-lsp7s" Sep 18 06:59:16.646824 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:59:16.646796 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzn6z\" (UniqueName: \"kubernetes.io/projected/45abc740-d22c-4fcc-af21-e75a01952cd6-kube-api-access-lzn6z\") pod \"payload-pre-processing-e2e-ipp-b-1672e6-b445c6b9d-lsp7s\" (UID: \"45abc740-d22c-4fcc-af21-e75a01952cd6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1672e6-b445c6b9d-lsp7s" Sep 18 06:59:16.646947 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:59:16.646843 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptjd6\" (UniqueName: \"kubernetes.io/projected/c1e71afc-e8f0-4bc8-b1b1-93d16f5c1641-kube-api-access-ptjd6\") pod \"payload-processing-e2e-ipp-b-1672e6-58bfdf58c7-7mvlb\" (UID: \"c1e71afc-e8f0-4bc8-b1b1-93d16f5c1641\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1672e6-58bfdf58c7-7mvlb" Sep 18 06:59:16.775793 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:59:16.775693 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1672e6-b445c6b9d-lsp7s" Sep 18 06:59:16.829613 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:59:16.829588 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1672e6-58bfdf58c7-7mvlb" Sep 18 06:59:17.049054 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:59:17.049020 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1672e6-b445c6b9d-lsp7s"] Sep 18 06:59:17.049925 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:59:17.049895 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45abc740_d22c_4fcc_af21_e75a01952cd6.slice/crio-c4449366e827ccbca34b588cf62281ec52a1dc12bd7f228db752d502a8c6b0e4 WatchSource:0}: Error finding container c4449366e827ccbca34b588cf62281ec52a1dc12bd7f228db752d502a8c6b0e4: Status 404 returned error can't find the container with id c4449366e827ccbca34b588cf62281ec52a1dc12bd7f228db752d502a8c6b0e4 Sep 18 06:59:17.113889 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:59:17.113866 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1672e6-58bfdf58c7-7mvlb"] Sep 18 06:59:17.115697 ip-10-0-142-104 kubenswrapper[2567]: W0918 06:59:17.115662 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1e71afc_e8f0_4bc8_b1b1_93d16f5c1641.slice/crio-08e2cbcf0bd7633bd2f22f4c142d167fda4262777d6a6b90d9e1986c98bcbbed WatchSource:0}: Error finding container 08e2cbcf0bd7633bd2f22f4c142d167fda4262777d6a6b90d9e1986c98bcbbed: Status 404 returned error can't find the container with id 08e2cbcf0bd7633bd2f22f4c142d167fda4262777d6a6b90d9e1986c98bcbbed Sep 18 06:59:17.928342 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:59:17.928300 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1672e6-b445c6b9d-lsp7s" event={"ID":"45abc740-d22c-4fcc-af21-e75a01952cd6","Type":"ContainerStarted","Data":"3ba4878b7f617d653394a9e8c6bab5c11f60f8bc8ba5cb6b1fa2372255ff4725"} Sep 18 06:59:17.928938 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:59:17.928384 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1672e6-b445c6b9d-lsp7s" event={"ID":"45abc740-d22c-4fcc-af21-e75a01952cd6","Type":"ContainerStarted","Data":"c4449366e827ccbca34b588cf62281ec52a1dc12bd7f228db752d502a8c6b0e4"} Sep 18 06:59:17.928938 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:59:17.928416 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1672e6-b445c6b9d-lsp7s" Sep 18 06:59:17.930090 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:59:17.930066 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1672e6-58bfdf58c7-7mvlb" event={"ID":"c1e71afc-e8f0-4bc8-b1b1-93d16f5c1641","Type":"ContainerStarted","Data":"51cf4fb122ba43f0387cdf1518f44262ec542fa6a4e22f783e00b2cca131ddb8"} Sep 18 06:59:17.930223 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:59:17.930114 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1672e6-58bfdf58c7-7mvlb" event={"ID":"c1e71afc-e8f0-4bc8-b1b1-93d16f5c1641","Type":"ContainerStarted","Data":"08e2cbcf0bd7633bd2f22f4c142d167fda4262777d6a6b90d9e1986c98bcbbed"} Sep 18 06:59:17.930223 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:59:17.930144 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-1672e6-58bfdf58c7-7mvlb" Sep 18 06:59:17.945289 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:59:17.945224 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1672e6-b445c6b9d-lsp7s" podStartSLOduration=1.945204841 podStartE2EDuration="1.945204841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:59:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:59:17.942789903 +0000 UTC m=+2219.682206078" watchObservedRunningTime="2026-09-18 06:59:17.945204841 +0000 UTC m=+2219.684621021" Sep 18 06:59:17.961555 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:59:17.961499 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-1672e6-58bfdf58c7-7mvlb" podStartSLOduration=1.961485159 podStartE2EDuration="1.961485159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:59:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:59:17.959866414 +0000 UTC m=+2219.699282629" watchObservedRunningTime="2026-09-18 06:59:17.961485159 +0000 UTC m=+2219.700901333" Sep 18 06:59:28.939420 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:59:28.939378 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-1672e6-58bfdf58c7-7mvlb" Sep 18 06:59:28.939819 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:59:28.939503 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1672e6-b445c6b9d-lsp7s" Sep 18 06:59:48.657153 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:59:48.657075 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-jbzm5_96a61b61-7de1-428e-9fb6-99d5534fb7db/payload-processing/0.log" Sep 18 06:59:48.771753 ip-10-0-142-104 kubenswrapper[2567]: I0918 06:59:48.771721 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-1672e6-58bfdf58c7-7mvlb_c1e71afc-e8f0-4bc8-b1b1-93d16f5c1641/payload-processing/0.log" Sep 18 07:00:00.134852 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:00.134815 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29828580-5krzm"] Sep 18 07:00:00.138550 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:00.138531 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a7bd58-29828580-2zpbw"] Sep 18 07:00:00.138689 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:00.138666 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828580-5krzm" Sep 18 07:00:00.143384 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:00.141573 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-88t7h\"" Sep 18 07:00:00.150664 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:00.150631 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1672e6-29828580-7sm2f"] Sep 18 07:00:00.151020 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:00.150993 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a7bd58-29828580-2zpbw" Sep 18 07:00:00.156504 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:00.156477 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a7bd58-29828580-2zpbw"] Sep 18 07:00:00.157124 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:00.157106 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1672e6-29828580-7sm2f"] Sep 18 07:00:00.157291 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:00.156579 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1672e6-29828580-7sm2f" Sep 18 07:00:00.158602 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:00.157831 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29828580-5krzm"] Sep 18 07:00:00.162910 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:00.162887 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828580-5krzm" Sep 18 07:00:00.171731 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:00.171711 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a7bd58-29828580-2zpbw" Sep 18 07:00:00.211042 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:00.210879 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1672e6-29828580-7sm2f" Sep 18 07:00:00.448642 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:00.448616 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29828580-5krzm"] Sep 18 07:00:00.448772 ip-10-0-142-104 kubenswrapper[2567]: W0918 07:00:00.448714 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40763e50_0e95_45b6_8df8_184fec55f673.slice/crio-13f78a93fee45c9007c69732cff887f9502dc3d43ad41df45b47cf54e38c061f WatchSource:0}: Error finding container 13f78a93fee45c9007c69732cff887f9502dc3d43ad41df45b47cf54e38c061f: Status 404 returned error can't find the container with id 13f78a93fee45c9007c69732cff887f9502dc3d43ad41df45b47cf54e38c061f Sep 18 07:00:00.463838 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:00.463810 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a7bd58-29828580-2zpbw"] Sep 18 07:00:00.467856 ip-10-0-142-104 kubenswrapper[2567]: W0918 07:00:00.467831 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20d096dd_8b74_45c6_a57e_2303eefa38dc.slice/crio-217219dc0d247376ec5a6fd6f049c15298338577d0da0e004d1cea6fa73186a9 WatchSource:0}: Error finding container 217219dc0d247376ec5a6fd6f049c15298338577d0da0e004d1cea6fa73186a9: Status 404 returned error can't find the container with id 217219dc0d247376ec5a6fd6f049c15298338577d0da0e004d1cea6fa73186a9 Sep 18 07:00:00.515421 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:00.515390 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1672e6-29828580-7sm2f"] Sep 18 07:00:00.516743 ip-10-0-142-104 kubenswrapper[2567]: W0918 07:00:00.516714 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44beed82_dd93_41ef_afc5_206cb8053e84.slice/crio-26d9596381f574b8d1dc5305122904d2c6920fd8cddd2895032acb1c69497905 WatchSource:0}: Error finding container 26d9596381f574b8d1dc5305122904d2c6920fd8cddd2895032acb1c69497905: Status 404 returned error can't find the container with id 26d9596381f574b8d1dc5305122904d2c6920fd8cddd2895032acb1c69497905 Sep 18 07:00:01.196470 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:01.196430 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a7bd58-29828580-2zpbw" event={"ID":"20d096dd-8b74-45c6-a57e-2303eefa38dc","Type":"ContainerStarted","Data":"a3dd1cb61de9b02718a2f28f94522a4917e6638305af15c408715141beb9c05f"} Sep 18 07:00:01.196874 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:01.196492 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a7bd58-29828580-2zpbw" event={"ID":"20d096dd-8b74-45c6-a57e-2303eefa38dc","Type":"ContainerStarted","Data":"217219dc0d247376ec5a6fd6f049c15298338577d0da0e004d1cea6fa73186a9"} Sep 18 07:00:01.198218 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:01.198190 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828580-5krzm" event={"ID":"40763e50-0e95-45b6-8df8-184fec55f673","Type":"ContainerStarted","Data":"9c177cb6949b43d41b017cf9e49d44329c274f3d1c560368df6662b7099305f7"} Sep 18 07:00:01.198360 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:01.198244 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828580-5krzm" event={"ID":"40763e50-0e95-45b6-8df8-184fec55f673","Type":"ContainerStarted","Data":"13f78a93fee45c9007c69732cff887f9502dc3d43ad41df45b47cf54e38c061f"} Sep 18 07:00:01.199910 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:01.199887 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1672e6-29828580-7sm2f" event={"ID":"44beed82-dd93-41ef-afc5-206cb8053e84","Type":"ContainerStarted","Data":"3d137e2bc5c86db8047c93a05b9c7686fc2b1a3cb2b8a8ce1e18865461a5e5ac"} Sep 18 07:00:01.200005 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:01.199927 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1672e6-29828580-7sm2f" event={"ID":"44beed82-dd93-41ef-afc5-206cb8053e84","Type":"ContainerStarted","Data":"26d9596381f574b8d1dc5305122904d2c6920fd8cddd2895032acb1c69497905"} Sep 18 07:00:01.213274 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:01.213230 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a7bd58-29828580-2zpbw" podStartSLOduration=1.213215476 podStartE2EDuration="1.213215476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 07:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 07:00:01.211659409 +0000 UTC m=+2262.951075601" watchObservedRunningTime="2026-09-18 07:00:01.213215476 +0000 UTC m=+2262.952631650" Sep 18 07:00:01.227601 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:01.227553 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1672e6-29828580-7sm2f" podStartSLOduration=1.2275386369999999 podStartE2EDuration="1.227538637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 07:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 07:00:01.224558966 +0000 UTC m=+2262.963975163" watchObservedRunningTime="2026-09-18 07:00:01.227538637 +0000 UTC m=+2262.966954831" Sep 18 07:00:01.238919 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:01.238870 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828580-5krzm" podStartSLOduration=1.238855376 podStartE2EDuration="1.238855376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 07:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 07:00:01.236946516 +0000 UTC m=+2262.976362714" watchObservedRunningTime="2026-09-18 07:00:01.238855376 +0000 UTC m=+2262.978271550" Sep 18 07:00:02.207863 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:02.207830 2567 generic.go:356] "Generic (PLEG): container finished" podID="44beed82-dd93-41ef-afc5-206cb8053e84" containerID="3d137e2bc5c86db8047c93a05b9c7686fc2b1a3cb2b8a8ce1e18865461a5e5ac" exitCode=0 Sep 18 07:00:02.208309 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:02.207918 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1672e6-29828580-7sm2f" event={"ID":"44beed82-dd93-41ef-afc5-206cb8053e84","Type":"ContainerDied","Data":"3d137e2bc5c86db8047c93a05b9c7686fc2b1a3cb2b8a8ce1e18865461a5e5ac"} Sep 18 07:00:02.209659 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:02.209635 2567 generic.go:356] "Generic (PLEG): container finished" podID="20d096dd-8b74-45c6-a57e-2303eefa38dc" containerID="a3dd1cb61de9b02718a2f28f94522a4917e6638305af15c408715141beb9c05f" exitCode=0 Sep 18 07:00:02.209775 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:02.209706 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a7bd58-29828580-2zpbw" event={"ID":"20d096dd-8b74-45c6-a57e-2303eefa38dc","Type":"ContainerDied","Data":"a3dd1cb61de9b02718a2f28f94522a4917e6638305af15c408715141beb9c05f"} Sep 18 07:00:02.211289 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:02.211268 2567 generic.go:356] "Generic (PLEG): container finished" podID="40763e50-0e95-45b6-8df8-184fec55f673" containerID="9c177cb6949b43d41b017cf9e49d44329c274f3d1c560368df6662b7099305f7" exitCode=0 Sep 18 07:00:02.211424 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:02.211320 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828580-5krzm" event={"ID":"40763e50-0e95-45b6-8df8-184fec55f673","Type":"ContainerDied","Data":"9c177cb6949b43d41b017cf9e49d44329c274f3d1c560368df6662b7099305f7"} Sep 18 07:00:03.390535 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:03.390507 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828580-5krzm" Sep 18 07:00:03.392898 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:03.392878 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1672e6-29828580-7sm2f" Sep 18 07:00:03.419088 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:03.419056 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a7bd58-29828580-2zpbw" Sep 18 07:00:04.225006 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:04.224971 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a7bd58-29828580-2zpbw" event={"ID":"20d096dd-8b74-45c6-a57e-2303eefa38dc","Type":"ContainerDied","Data":"217219dc0d247376ec5a6fd6f049c15298338577d0da0e004d1cea6fa73186a9"} Sep 18 07:00:04.225222 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:04.225038 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="217219dc0d247376ec5a6fd6f049c15298338577d0da0e004d1cea6fa73186a9" Sep 18 07:00:04.225222 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:04.224976 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a7bd58-29828580-2zpbw" Sep 18 07:00:04.226512 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:04.226486 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828580-5krzm" Sep 18 07:00:04.226633 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:04.226538 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828580-5krzm" event={"ID":"40763e50-0e95-45b6-8df8-184fec55f673","Type":"ContainerDied","Data":"13f78a93fee45c9007c69732cff887f9502dc3d43ad41df45b47cf54e38c061f"} Sep 18 07:00:04.226633 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:04.226573 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13f78a93fee45c9007c69732cff887f9502dc3d43ad41df45b47cf54e38c061f" Sep 18 07:00:04.228089 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:04.228070 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1672e6-29828580-7sm2f" Sep 18 07:00:04.228192 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:04.228108 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1672e6-29828580-7sm2f" event={"ID":"44beed82-dd93-41ef-afc5-206cb8053e84","Type":"ContainerDied","Data":"26d9596381f574b8d1dc5305122904d2c6920fd8cddd2895032acb1c69497905"} Sep 18 07:00:04.228192 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:04.228158 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26d9596381f574b8d1dc5305122904d2c6920fd8cddd2895032acb1c69497905" Sep 18 07:00:19.557434 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:19.557403 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-a7bd58-6999dbc775-qcqlf_98d614d7-ff85-4f64-b4a2-d85549dd05af/payload-processing/0.log" Sep 18 07:00:19.680881 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:19.680851 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-jbzm5_96a61b61-7de1-428e-9fb6-99d5534fb7db/payload-processing/0.log" Sep 18 07:00:19.793956 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:19.793923 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-1672e6-58bfdf58c7-7mvlb_c1e71afc-e8f0-4bc8-b1b1-93d16f5c1641/payload-processing/0.log" Sep 18 07:00:19.905382 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:19.905350 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-1672e6-58bfdf58c7-7mvlb_c1e71afc-e8f0-4bc8-b1b1-93d16f5c1641/payload-processing/0.log" Sep 18 07:00:20.426716 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:20.426678 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp"] Sep 18 07:00:20.427246 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:20.427225 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40763e50-0e95-45b6-8df8-184fec55f673" containerName="cleanup" Sep 18 07:00:20.427246 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:20.427245 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="40763e50-0e95-45b6-8df8-184fec55f673" containerName="cleanup" Sep 18 07:00:20.427466 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:20.427260 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20d096dd-8b74-45c6-a57e-2303eefa38dc" containerName="cleanup" Sep 18 07:00:20.427466 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:20.427266 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="20d096dd-8b74-45c6-a57e-2303eefa38dc" containerName="cleanup" Sep 18 07:00:20.427466 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:20.427284 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44beed82-dd93-41ef-afc5-206cb8053e84" containerName="cleanup" Sep 18 07:00:20.427466 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:20.427290 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="44beed82-dd93-41ef-afc5-206cb8053e84" containerName="cleanup" Sep 18 07:00:20.427466 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:20.427359 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="20d096dd-8b74-45c6-a57e-2303eefa38dc" containerName="cleanup" Sep 18 07:00:20.427466 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:20.427382 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="44beed82-dd93-41ef-afc5-206cb8053e84" containerName="cleanup" Sep 18 07:00:20.427466 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:20.427390 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="40763e50-0e95-45b6-8df8-184fec55f673" containerName="cleanup" Sep 18 07:00:20.430653 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:20.430631 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp" Sep 18 07:00:20.433405 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:20.433360 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-75ce90-openshift-default-dockercfg-lb5w4\"" Sep 18 07:00:20.448097 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:20.448069 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp"] Sep 18 07:00:20.576485 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:20.576451 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65ba9bd5-6fd8-489a-b30f-2481ca8be065-istio-envoy\") pod \"e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp\" (UID: \"65ba9bd5-6fd8-489a-b30f-2481ca8be065\") " pod="openshift-ingress/e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp" Sep 18 07:00:20.576485 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:20.576484 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ba9bd5-6fd8-489a-b30f-2481ca8be065-credential-socket\") pod \"e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp\" (UID: \"65ba9bd5-6fd8-489a-b30f-2481ca8be065\") " pod="openshift-ingress/e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp" Sep 18 07:00:20.576889 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:20.576553 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65ba9bd5-6fd8-489a-b30f-2481ca8be065-istiod-ca-cert\") pod \"e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp\" (UID: \"65ba9bd5-6fd8-489a-b30f-2481ca8be065\") " pod="openshift-ingress/e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp" Sep 18 07:00:20.576889 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:20.576578 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65ba9bd5-6fd8-489a-b30f-2481ca8be065-istio-podinfo\") pod \"e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp\" (UID: \"65ba9bd5-6fd8-489a-b30f-2481ca8be065\") " pod="openshift-ingress/e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp" Sep 18 07:00:20.576889 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:20.576593 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65ba9bd5-6fd8-489a-b30f-2481ca8be065-istio-token\") pod \"e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp\" (UID: \"65ba9bd5-6fd8-489a-b30f-2481ca8be065\") " pod="openshift-ingress/e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp" Sep 18 07:00:20.576889 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:20.576667 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmlxq\" (UniqueName: \"kubernetes.io/projected/65ba9bd5-6fd8-489a-b30f-2481ca8be065-kube-api-access-nmlxq\") pod \"e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp\" (UID: \"65ba9bd5-6fd8-489a-b30f-2481ca8be065\") " pod="openshift-ingress/e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp" Sep 18 07:00:20.576889 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:20.576722 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ba9bd5-6fd8-489a-b30f-2481ca8be065-workload-socket\") pod \"e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp\" (UID: \"65ba9bd5-6fd8-489a-b30f-2481ca8be065\") " pod="openshift-ingress/e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp" Sep 18 07:00:20.576889 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:20.576752 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65ba9bd5-6fd8-489a-b30f-2481ca8be065-istio-data\") pod \"e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp\" (UID: \"65ba9bd5-6fd8-489a-b30f-2481ca8be065\") " pod="openshift-ingress/e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp" Sep 18 07:00:20.576889 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:20.576813 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65ba9bd5-6fd8-489a-b30f-2481ca8be065-workload-certs\") pod \"e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp\" (UID: \"65ba9bd5-6fd8-489a-b30f-2481ca8be065\") " pod="openshift-ingress/e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp" Sep 18 07:00:20.677494 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:20.677403 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ba9bd5-6fd8-489a-b30f-2481ca8be065-workload-socket\") pod \"e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp\" (UID: \"65ba9bd5-6fd8-489a-b30f-2481ca8be065\") " pod="openshift-ingress/e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp" Sep 18 07:00:20.677494 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:20.677448 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65ba9bd5-6fd8-489a-b30f-2481ca8be065-istio-data\") pod \"e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp\" (UID: \"65ba9bd5-6fd8-489a-b30f-2481ca8be065\") " pod="openshift-ingress/e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp" Sep 18 07:00:20.677494 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:20.677495 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65ba9bd5-6fd8-489a-b30f-2481ca8be065-workload-certs\") pod \"e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp\" (UID: \"65ba9bd5-6fd8-489a-b30f-2481ca8be065\") " pod="openshift-ingress/e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp" Sep 18 07:00:20.677786 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:20.677526 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65ba9bd5-6fd8-489a-b30f-2481ca8be065-istio-envoy\") pod \"e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp\" (UID: \"65ba9bd5-6fd8-489a-b30f-2481ca8be065\") " pod="openshift-ingress/e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp" Sep 18 07:00:20.677786 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:20.677550 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ba9bd5-6fd8-489a-b30f-2481ca8be065-credential-socket\") pod \"e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp\" (UID: \"65ba9bd5-6fd8-489a-b30f-2481ca8be065\") " pod="openshift-ingress/e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp" Sep 18 07:00:20.677786 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:20.677614 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65ba9bd5-6fd8-489a-b30f-2481ca8be065-istiod-ca-cert\") pod \"e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp\" (UID: \"65ba9bd5-6fd8-489a-b30f-2481ca8be065\") " pod="openshift-ingress/e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp" Sep 18 07:00:20.677786 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:20.677642 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65ba9bd5-6fd8-489a-b30f-2481ca8be065-istio-podinfo\") pod \"e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp\" (UID: \"65ba9bd5-6fd8-489a-b30f-2481ca8be065\") " pod="openshift-ingress/e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp" Sep 18 07:00:20.677786 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:20.677669 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65ba9bd5-6fd8-489a-b30f-2481ca8be065-istio-token\") pod \"e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp\" (UID: \"65ba9bd5-6fd8-489a-b30f-2481ca8be065\") " pod="openshift-ingress/e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp" Sep 18 07:00:20.677786 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:20.677724 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmlxq\" (UniqueName: \"kubernetes.io/projected/65ba9bd5-6fd8-489a-b30f-2481ca8be065-kube-api-access-nmlxq\") pod \"e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp\" (UID: \"65ba9bd5-6fd8-489a-b30f-2481ca8be065\") " pod="openshift-ingress/e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp" Sep 18 07:00:20.678112 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:20.677833 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ba9bd5-6fd8-489a-b30f-2481ca8be065-workload-socket\") pod \"e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp\" (UID: \"65ba9bd5-6fd8-489a-b30f-2481ca8be065\") " pod="openshift-ingress/e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp" Sep 18 07:00:20.678194 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:20.678160 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65ba9bd5-6fd8-489a-b30f-2481ca8be065-workload-certs\") pod \"e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp\" (UID: \"65ba9bd5-6fd8-489a-b30f-2481ca8be065\") " pod="openshift-ingress/e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp" Sep 18 07:00:20.678263 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:20.678201 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65ba9bd5-6fd8-489a-b30f-2481ca8be065-istio-data\") pod \"e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp\" (UID: \"65ba9bd5-6fd8-489a-b30f-2481ca8be065\") " pod="openshift-ingress/e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp" Sep 18 07:00:20.678321 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:20.678280 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ba9bd5-6fd8-489a-b30f-2481ca8be065-credential-socket\") pod \"e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp\" (UID: \"65ba9bd5-6fd8-489a-b30f-2481ca8be065\") " pod="openshift-ingress/e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp" Sep 18 07:00:20.678398 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:20.678315 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65ba9bd5-6fd8-489a-b30f-2481ca8be065-istiod-ca-cert\") pod \"e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp\" (UID: \"65ba9bd5-6fd8-489a-b30f-2481ca8be065\") " pod="openshift-ingress/e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp" Sep 18 07:00:20.680592 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:20.680568 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65ba9bd5-6fd8-489a-b30f-2481ca8be065-istio-envoy\") pod \"e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp\" (UID: \"65ba9bd5-6fd8-489a-b30f-2481ca8be065\") " pod="openshift-ingress/e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp" Sep 18 07:00:20.680768 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:20.680751 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65ba9bd5-6fd8-489a-b30f-2481ca8be065-istio-podinfo\") pod \"e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp\" (UID: \"65ba9bd5-6fd8-489a-b30f-2481ca8be065\") " pod="openshift-ingress/e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp" Sep 18 07:00:20.685609 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:20.685586 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65ba9bd5-6fd8-489a-b30f-2481ca8be065-istio-token\") pod \"e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp\" (UID: \"65ba9bd5-6fd8-489a-b30f-2481ca8be065\") " pod="openshift-ingress/e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp" Sep 18 07:00:20.685788 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:20.685772 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmlxq\" (UniqueName: \"kubernetes.io/projected/65ba9bd5-6fd8-489a-b30f-2481ca8be065-kube-api-access-nmlxq\") pod \"e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp\" (UID: \"65ba9bd5-6fd8-489a-b30f-2481ca8be065\") " pod="openshift-ingress/e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp" Sep 18 07:00:20.744143 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:20.744103 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp" Sep 18 07:00:20.958152 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:20.958065 2567 scope.go:117] "RemoveContainer" containerID="d5d37d5657f3346fcfdd6a6754081c9be8d7fc161c8051d32603f389b3ed99ee" Sep 18 07:00:20.967604 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:20.967581 2567 scope.go:117] "RemoveContainer" containerID="e78d704256aab946f646b5f6eddc09f959ed0b3a3250f9f4a46fb1ee9b1172f8" Sep 18 07:00:20.976829 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:20.976808 2567 scope.go:117] "RemoveContainer" containerID="21475b459c3abf4277cb10483c8ee028d407ff0fe3350a355ca75bb4fb0c2736" Sep 18 07:00:21.027235 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:21.027202 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp"] Sep 18 07:00:21.028645 ip-10-0-142-104 kubenswrapper[2567]: W0918 07:00:21.028608 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65ba9bd5_6fd8_489a_b30f_2481ca8be065.slice/crio-6861b2a41e771bd3aecc2137107f0ca80783a97617b5d2b9fa70f6856a2209d1 WatchSource:0}: Error finding container 6861b2a41e771bd3aecc2137107f0ca80783a97617b5d2b9fa70f6856a2209d1: Status 404 returned error can't find the container with id 6861b2a41e771bd3aecc2137107f0ca80783a97617b5d2b9fa70f6856a2209d1 Sep 18 07:00:21.030952 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:21.030919 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 07:00:21.031053 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:21.031001 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 07:00:21.031053 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:21.031043 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 07:00:21.335786 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:21.335754 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp" event={"ID":"65ba9bd5-6fd8-489a-b30f-2481ca8be065","Type":"ContainerStarted","Data":"d3c966397f10bdefdb6513ccfeff8d55d3a69fd551fa62da98843aafa39f14c0"} Sep 18 07:00:21.335978 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:21.335807 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp" event={"ID":"65ba9bd5-6fd8-489a-b30f-2481ca8be065","Type":"ContainerStarted","Data":"6861b2a41e771bd3aecc2137107f0ca80783a97617b5d2b9fa70f6856a2209d1"} Sep 18 07:00:21.357033 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:21.356978 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp" podStartSLOduration=1.356961668 podStartE2EDuration="1.356961668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 07:00:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 07:00:21.355117618 +0000 UTC m=+2283.094533832" watchObservedRunningTime="2026-09-18 07:00:21.356961668 +0000 UTC m=+2283.096377842" Sep 18 07:00:21.744270 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:21.744176 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp" Sep 18 07:00:22.749616 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:22.749583 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp" Sep 18 07:00:23.349155 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:23.349116 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp" Sep 18 07:00:23.351048 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:23.351017 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp" Sep 18 07:00:26.750487 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:26.750448 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-75ce90-86f8c79549-29njf"] Sep 18 07:00:26.754998 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:26.754973 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-75ce90-86f8c79549-29njf" Sep 18 07:00:26.758270 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:26.758240 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-75ce90\"" Sep 18 07:00:26.758446 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:26.758312 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-75ce90-dockercfg-fsnpx\"" Sep 18 07:00:26.768571 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:26.768545 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-75ce90-86f8c79549-29njf"] Sep 18 07:00:26.842314 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:26.842272 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd498850-dd87-4a85-8c92-a4e2c3dde8d9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-75ce90-86f8c79549-29njf\" (UID: \"fd498850-dd87-4a85-8c92-a4e2c3dde8d9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-75ce90-86f8c79549-29njf" Sep 18 07:00:26.842757 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:26.842718 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shvtr\" (UniqueName: \"kubernetes.io/projected/fd498850-dd87-4a85-8c92-a4e2c3dde8d9-kube-api-access-shvtr\") pod \"payload-processing-e2e-ipp-cleanup-75ce90-86f8c79549-29njf\" (UID: \"fd498850-dd87-4a85-8c92-a4e2c3dde8d9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-75ce90-86f8c79549-29njf" Sep 18 07:00:26.944119 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:26.944088 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd498850-dd87-4a85-8c92-a4e2c3dde8d9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-75ce90-86f8c79549-29njf\" (UID: \"fd498850-dd87-4a85-8c92-a4e2c3dde8d9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-75ce90-86f8c79549-29njf" Sep 18 07:00:26.944286 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:26.944185 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shvtr\" (UniqueName: \"kubernetes.io/projected/fd498850-dd87-4a85-8c92-a4e2c3dde8d9-kube-api-access-shvtr\") pod \"payload-processing-e2e-ipp-cleanup-75ce90-86f8c79549-29njf\" (UID: \"fd498850-dd87-4a85-8c92-a4e2c3dde8d9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-75ce90-86f8c79549-29njf" Sep 18 07:00:26.944901 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:26.944875 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd498850-dd87-4a85-8c92-a4e2c3dde8d9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-75ce90-86f8c79549-29njf\" (UID: \"fd498850-dd87-4a85-8c92-a4e2c3dde8d9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-75ce90-86f8c79549-29njf" Sep 18 07:00:26.951668 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:26.951644 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shvtr\" (UniqueName: \"kubernetes.io/projected/fd498850-dd87-4a85-8c92-a4e2c3dde8d9-kube-api-access-shvtr\") pod \"payload-processing-e2e-ipp-cleanup-75ce90-86f8c79549-29njf\" (UID: \"fd498850-dd87-4a85-8c92-a4e2c3dde8d9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-75ce90-86f8c79549-29njf" Sep 18 07:00:27.067921 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:27.067831 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-75ce90-86f8c79549-29njf" Sep 18 07:00:27.342750 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:27.342722 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-75ce90-86f8c79549-29njf"] Sep 18 07:00:27.344160 ip-10-0-142-104 kubenswrapper[2567]: W0918 07:00:27.344131 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd498850_dd87_4a85_8c92_a4e2c3dde8d9.slice/crio-a8a4dc50710563a22372acb070dc703d17d11a037de8231eb9c05622e96bd845 WatchSource:0}: Error finding container a8a4dc50710563a22372acb070dc703d17d11a037de8231eb9c05622e96bd845: Status 404 returned error can't find the container with id a8a4dc50710563a22372acb070dc703d17d11a037de8231eb9c05622e96bd845 Sep 18 07:00:27.376353 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:27.376316 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-75ce90-86f8c79549-29njf" event={"ID":"fd498850-dd87-4a85-8c92-a4e2c3dde8d9","Type":"ContainerStarted","Data":"a8a4dc50710563a22372acb070dc703d17d11a037de8231eb9c05622e96bd845"} Sep 18 07:00:28.384974 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:28.384937 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-75ce90-86f8c79549-29njf" event={"ID":"fd498850-dd87-4a85-8c92-a4e2c3dde8d9","Type":"ContainerStarted","Data":"7f98e3cc9c621d5ce8585a9569e872fd21adbc6471ba76e0e2a6b1aa93d4ae21"} Sep 18 07:00:28.385484 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:28.385022 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-75ce90-86f8c79549-29njf" Sep 18 07:00:28.404827 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:28.404771 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-75ce90-86f8c79549-29njf" podStartSLOduration=2.404752568 podStartE2EDuration="2.404752568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 07:00:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 07:00:28.401741384 +0000 UTC m=+2290.141157650" watchObservedRunningTime="2026-09-18 07:00:28.404752568 +0000 UTC m=+2290.144168742" Sep 18 07:00:39.393038 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:39.393006 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-75ce90-86f8c79549-29njf" Sep 18 07:00:43.505246 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:43.505209 2567 generic.go:356] "Generic (PLEG): container finished" podID="65ba9bd5-6fd8-489a-b30f-2481ca8be065" containerID="d3c966397f10bdefdb6513ccfeff8d55d3a69fd551fa62da98843aafa39f14c0" exitCode=0 Sep 18 07:00:43.505720 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:43.505281 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp" event={"ID":"65ba9bd5-6fd8-489a-b30f-2481ca8be065","Type":"ContainerDied","Data":"d3c966397f10bdefdb6513ccfeff8d55d3a69fd551fa62da98843aafa39f14c0"} Sep 18 07:00:43.505848 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:43.505830 2567 scope.go:117] "RemoveContainer" containerID="d3c966397f10bdefdb6513ccfeff8d55d3a69fd551fa62da98843aafa39f14c0" Sep 18 07:00:43.506978 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:43.506948 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 07:00:43.507068 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:43.507015 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 07:00:43.507068 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:43.507045 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 18 07:00:44.514102 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:44.514067 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp" event={"ID":"65ba9bd5-6fd8-489a-b30f-2481ca8be065","Type":"ContainerStarted","Data":"0202dfab41b7ba25be88fdabe40fb70f500d7243bf23ba4a7468e9263dacdd84"} Sep 18 07:00:44.745207 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:44.745165 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp" Sep 18 07:00:44.745459 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:44.745434 2567 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Sep 18 07:00:44.745524 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:44.745489 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp" podUID="65ba9bd5-6fd8-489a-b30f-2481ca8be065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Sep 18 07:00:45.744780 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:45.744745 2567 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Sep 18 07:00:45.745179 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:45.744802 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp" podUID="65ba9bd5-6fd8-489a-b30f-2481ca8be065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Sep 18 07:00:46.744725 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:46.744683 2567 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Sep 18 07:00:46.744912 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:46.744759 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp" podUID="65ba9bd5-6fd8-489a-b30f-2481ca8be065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Sep 18 07:00:47.744957 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:47.744918 2567 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Sep 18 07:00:47.745403 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:47.744974 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp" podUID="65ba9bd5-6fd8-489a-b30f-2481ca8be065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Sep 18 07:00:48.745436 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:48.745361 2567 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Sep 18 07:00:48.745929 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:48.745457 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp" podUID="65ba9bd5-6fd8-489a-b30f-2481ca8be065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Sep 18 07:00:49.749317 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:49.749282 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp" Sep 18 07:00:49.749852 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:49.749709 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp" Sep 18 07:00:49.750405 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:49.750355 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp" Sep 18 07:00:53.974335 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:53.974299 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-75ce90-5qx77"] Sep 18 07:00:53.977035 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:53.977017 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-75ce90-5qx77" Sep 18 07:00:53.979663 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:53.979640 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-88t7h\"" Sep 18 07:00:53.979785 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:53.979640 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 18 07:00:53.985766 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:53.985416 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-75ce90-5qx77"] Sep 18 07:00:54.147258 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:54.147213 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/417843e7-102e-47d5-adbc-5e38bd096a62-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-75ce90-5qx77\" (UID: \"417843e7-102e-47d5-adbc-5e38bd096a62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-75ce90-5qx77" Sep 18 07:00:54.248762 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:54.248675 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/417843e7-102e-47d5-adbc-5e38bd096a62-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-75ce90-5qx77\" (UID: \"417843e7-102e-47d5-adbc-5e38bd096a62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-75ce90-5qx77" Sep 18 07:00:54.249303 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:54.249283 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/417843e7-102e-47d5-adbc-5e38bd096a62-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-75ce90-5qx77\" (UID: \"417843e7-102e-47d5-adbc-5e38bd096a62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-75ce90-5qx77" Sep 18 07:00:54.291826 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:54.291792 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-75ce90-5qx77" Sep 18 07:00:54.570133 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:54.570090 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-75ce90-5qx77"] Sep 18 07:00:54.571161 ip-10-0-142-104 kubenswrapper[2567]: W0918 07:00:54.571137 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod417843e7_102e_47d5_adbc_5e38bd096a62.slice/crio-e311f3b030fe84f82f2fb31ae52af3245d582562a7c09f55c6cd484467dadead WatchSource:0}: Error finding container e311f3b030fe84f82f2fb31ae52af3245d582562a7c09f55c6cd484467dadead: Status 404 returned error can't find the container with id e311f3b030fe84f82f2fb31ae52af3245d582562a7c09f55c6cd484467dadead Sep 18 07:00:55.585802 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:55.585770 2567 generic.go:356] "Generic (PLEG): container finished" podID="417843e7-102e-47d5-adbc-5e38bd096a62" containerID="2cb7b5f147b09a8485c4e995f188bac7215ab7104cecc936063f04b041d617d7" exitCode=0 Sep 18 07:00:55.586197 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:55.585867 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-75ce90-5qx77" event={"ID":"417843e7-102e-47d5-adbc-5e38bd096a62","Type":"ContainerDied","Data":"2cb7b5f147b09a8485c4e995f188bac7215ab7104cecc936063f04b041d617d7"} Sep 18 07:00:55.586197 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:55.585916 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-75ce90-5qx77" event={"ID":"417843e7-102e-47d5-adbc-5e38bd096a62","Type":"ContainerStarted","Data":"e311f3b030fe84f82f2fb31ae52af3245d582562a7c09f55c6cd484467dadead"} Sep 18 07:00:56.731739 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:56.731715 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-75ce90-5qx77" Sep 18 07:00:56.876160 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:56.876058 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/417843e7-102e-47d5-adbc-5e38bd096a62-maas-api-service-ca\") pod \"417843e7-102e-47d5-adbc-5e38bd096a62\" (UID: \"417843e7-102e-47d5-adbc-5e38bd096a62\") " Sep 18 07:00:56.876481 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:56.876459 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/417843e7-102e-47d5-adbc-5e38bd096a62-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "417843e7-102e-47d5-adbc-5e38bd096a62" (UID: "417843e7-102e-47d5-adbc-5e38bd096a62"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 07:00:56.977798 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:56.977763 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/417843e7-102e-47d5-adbc-5e38bd096a62-maas-api-service-ca\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 07:00:57.608075 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:57.608049 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-75ce90-5qx77" Sep 18 07:00:57.608241 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:57.608046 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-75ce90-5qx77" event={"ID":"417843e7-102e-47d5-adbc-5e38bd096a62","Type":"ContainerDied","Data":"e311f3b030fe84f82f2fb31ae52af3245d582562a7c09f55c6cd484467dadead"} Sep 18 07:00:57.608241 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:00:57.608167 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e311f3b030fe84f82f2fb31ae52af3245d582562a7c09f55c6cd484467dadead" Sep 18 07:01:04.928958 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:04.928868 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-75ce90-86f8c79549-29njf"] Sep 18 07:01:04.929393 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:04.929093 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-75ce90-86f8c79549-29njf" podUID="fd498850-dd87-4a85-8c92-a4e2c3dde8d9" containerName="payload-processing" containerID="cri-o://7f98e3cc9c621d5ce8585a9569e872fd21adbc6471ba76e0e2a6b1aa93d4ae21" gracePeriod=30 Sep 18 07:01:05.109823 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:05.109788 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-75ce90-5qx77"] Sep 18 07:01:05.112609 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:05.112584 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-75ce90-5qx77"] Sep 18 07:01:05.192823 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:05.192754 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-75ce90-86f8c79549-29njf" Sep 18 07:01:05.256104 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:05.256057 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd498850-dd87-4a85-8c92-a4e2c3dde8d9-plugins-config-volume\") pod \"fd498850-dd87-4a85-8c92-a4e2c3dde8d9\" (UID: \"fd498850-dd87-4a85-8c92-a4e2c3dde8d9\") " Sep 18 07:01:05.256351 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:05.256123 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-shvtr\" (UniqueName: \"kubernetes.io/projected/fd498850-dd87-4a85-8c92-a4e2c3dde8d9-kube-api-access-shvtr\") pod \"fd498850-dd87-4a85-8c92-a4e2c3dde8d9\" (UID: \"fd498850-dd87-4a85-8c92-a4e2c3dde8d9\") " Sep 18 07:01:05.256670 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:05.256635 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd498850-dd87-4a85-8c92-a4e2c3dde8d9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fd498850-dd87-4a85-8c92-a4e2c3dde8d9" (UID: "fd498850-dd87-4a85-8c92-a4e2c3dde8d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 07:01:05.259092 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:05.259030 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd498850-dd87-4a85-8c92-a4e2c3dde8d9-kube-api-access-shvtr" (OuterVolumeSpecName: "kube-api-access-shvtr") pod "fd498850-dd87-4a85-8c92-a4e2c3dde8d9" (UID: "fd498850-dd87-4a85-8c92-a4e2c3dde8d9"). InnerVolumeSpecName "kube-api-access-shvtr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 07:01:05.357412 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:05.357354 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd498850-dd87-4a85-8c92-a4e2c3dde8d9-plugins-config-volume\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 07:01:05.357412 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:05.357405 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shvtr\" (UniqueName: \"kubernetes.io/projected/fd498850-dd87-4a85-8c92-a4e2c3dde8d9-kube-api-access-shvtr\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 07:01:05.657647 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:05.657605 2567 generic.go:356] "Generic (PLEG): container finished" podID="fd498850-dd87-4a85-8c92-a4e2c3dde8d9" containerID="7f98e3cc9c621d5ce8585a9569e872fd21adbc6471ba76e0e2a6b1aa93d4ae21" exitCode=0 Sep 18 07:01:05.657840 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:05.657638 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-75ce90-86f8c79549-29njf" event={"ID":"fd498850-dd87-4a85-8c92-a4e2c3dde8d9","Type":"ContainerDied","Data":"7f98e3cc9c621d5ce8585a9569e872fd21adbc6471ba76e0e2a6b1aa93d4ae21"} Sep 18 07:01:05.657840 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:05.657660 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-75ce90-86f8c79549-29njf" Sep 18 07:01:05.657840 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:05.657704 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-75ce90-86f8c79549-29njf" event={"ID":"fd498850-dd87-4a85-8c92-a4e2c3dde8d9","Type":"ContainerDied","Data":"a8a4dc50710563a22372acb070dc703d17d11a037de8231eb9c05622e96bd845"} Sep 18 07:01:05.657840 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:05.657728 2567 scope.go:117] "RemoveContainer" containerID="7f98e3cc9c621d5ce8585a9569e872fd21adbc6471ba76e0e2a6b1aa93d4ae21" Sep 18 07:01:05.667771 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:05.667751 2567 scope.go:117] "RemoveContainer" containerID="7f98e3cc9c621d5ce8585a9569e872fd21adbc6471ba76e0e2a6b1aa93d4ae21" Sep 18 07:01:05.668075 ip-10-0-142-104 kubenswrapper[2567]: E0918 07:01:05.668055 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f98e3cc9c621d5ce8585a9569e872fd21adbc6471ba76e0e2a6b1aa93d4ae21\": container with ID starting with 7f98e3cc9c621d5ce8585a9569e872fd21adbc6471ba76e0e2a6b1aa93d4ae21 not found: ID does not exist" containerID="7f98e3cc9c621d5ce8585a9569e872fd21adbc6471ba76e0e2a6b1aa93d4ae21" Sep 18 07:01:05.668126 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:05.668084 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f98e3cc9c621d5ce8585a9569e872fd21adbc6471ba76e0e2a6b1aa93d4ae21"} err="failed to get container status \"7f98e3cc9c621d5ce8585a9569e872fd21adbc6471ba76e0e2a6b1aa93d4ae21\": rpc error: code = NotFound desc = could not find container \"7f98e3cc9c621d5ce8585a9569e872fd21adbc6471ba76e0e2a6b1aa93d4ae21\": container with ID starting with 7f98e3cc9c621d5ce8585a9569e872fd21adbc6471ba76e0e2a6b1aa93d4ae21 not found: ID does not exist" Sep 18 07:01:05.680695 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:05.680653 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-75ce90-86f8c79549-29njf"] Sep 18 07:01:05.683556 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:05.683536 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-75ce90-86f8c79549-29njf"] Sep 18 07:01:06.447461 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:06.447420 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="417843e7-102e-47d5-adbc-5e38bd096a62" path="/var/lib/kubelet/pods/417843e7-102e-47d5-adbc-5e38bd096a62/volumes" Sep 18 07:01:06.447866 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:06.447852 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd498850-dd87-4a85-8c92-a4e2c3dde8d9" path="/var/lib/kubelet/pods/fd498850-dd87-4a85-8c92-a4e2c3dde8d9/volumes" Sep 18 07:01:11.374306 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:11.374269 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp"] Sep 18 07:01:11.374828 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:11.374508 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp" podUID="65ba9bd5-6fd8-489a-b30f-2481ca8be065" containerName="istio-proxy" containerID="cri-o://0202dfab41b7ba25be88fdabe40fb70f500d7243bf23ba4a7468e9263dacdd84" gracePeriod=30 Sep 18 07:01:14.853832 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:14.853790 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a7bd58-hr9zc"] Sep 18 07:01:14.854327 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:14.854265 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd498850-dd87-4a85-8c92-a4e2c3dde8d9" containerName="payload-processing" Sep 18 07:01:14.854327 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:14.854277 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd498850-dd87-4a85-8c92-a4e2c3dde8d9" containerName="payload-processing" Sep 18 07:01:14.854327 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:14.854294 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="417843e7-102e-47d5-adbc-5e38bd096a62" containerName="revoke-keys" Sep 18 07:01:14.854327 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:14.854300 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="417843e7-102e-47d5-adbc-5e38bd096a62" containerName="revoke-keys" Sep 18 07:01:14.854504 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:14.854358 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd498850-dd87-4a85-8c92-a4e2c3dde8d9" containerName="payload-processing" Sep 18 07:01:14.854504 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:14.854384 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="417843e7-102e-47d5-adbc-5e38bd096a62" containerName="revoke-keys" Sep 18 07:01:14.857549 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:14.857534 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a7bd58-hr9zc" Sep 18 07:01:14.860390 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:14.860339 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 18 07:01:14.860546 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:14.860412 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-88t7h\"" Sep 18 07:01:14.864322 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:14.864293 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a7bd58-hr9zc"] Sep 18 07:01:14.955246 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:14.955208 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a10adc9-1460-481c-9662-99112f5fd7ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a7bd58-hr9zc\" (UID: \"6a10adc9-1460-481c-9662-99112f5fd7ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a7bd58-hr9zc" Sep 18 07:01:15.056018 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:15.055977 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a10adc9-1460-481c-9662-99112f5fd7ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a7bd58-hr9zc\" (UID: \"6a10adc9-1460-481c-9662-99112f5fd7ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a7bd58-hr9zc" Sep 18 07:01:15.056682 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:15.056660 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a10adc9-1460-481c-9662-99112f5fd7ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a7bd58-hr9zc\" (UID: \"6a10adc9-1460-481c-9662-99112f5fd7ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a7bd58-hr9zc" Sep 18 07:01:15.173077 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:15.172984 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a7bd58-hr9zc" Sep 18 07:01:15.446042 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:15.445417 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a7bd58-hr9zc"] Sep 18 07:01:15.726302 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:15.726209 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a7bd58-hr9zc" event={"ID":"6a10adc9-1460-481c-9662-99112f5fd7ed","Type":"ContainerStarted","Data":"0da941467d4a7a107cf585f66af7a74ddf6c786a4dafac88119c1aa4a065ac01"} Sep 18 07:01:16.637377 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:16.637338 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp" Sep 18 07:01:16.734003 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:16.733966 2567 generic.go:356] "Generic (PLEG): container finished" podID="6a10adc9-1460-481c-9662-99112f5fd7ed" containerID="4a4923753ba3117596d1991269a2b18a38afb712c7dad5dbeabc5226932417b2" exitCode=0 Sep 18 07:01:16.734187 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:16.734052 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a7bd58-hr9zc" event={"ID":"6a10adc9-1460-481c-9662-99112f5fd7ed","Type":"ContainerDied","Data":"4a4923753ba3117596d1991269a2b18a38afb712c7dad5dbeabc5226932417b2"} Sep 18 07:01:16.735847 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:16.735825 2567 generic.go:356] "Generic (PLEG): container finished" podID="65ba9bd5-6fd8-489a-b30f-2481ca8be065" containerID="0202dfab41b7ba25be88fdabe40fb70f500d7243bf23ba4a7468e9263dacdd84" exitCode=0 Sep 18 07:01:16.735970 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:16.735848 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp" event={"ID":"65ba9bd5-6fd8-489a-b30f-2481ca8be065","Type":"ContainerDied","Data":"0202dfab41b7ba25be88fdabe40fb70f500d7243bf23ba4a7468e9263dacdd84"} Sep 18 07:01:16.735970 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:16.735870 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp" Sep 18 07:01:16.735970 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:16.735894 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp" event={"ID":"65ba9bd5-6fd8-489a-b30f-2481ca8be065","Type":"ContainerDied","Data":"6861b2a41e771bd3aecc2137107f0ca80783a97617b5d2b9fa70f6856a2209d1"} Sep 18 07:01:16.735970 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:16.735920 2567 scope.go:117] "RemoveContainer" containerID="0202dfab41b7ba25be88fdabe40fb70f500d7243bf23ba4a7468e9263dacdd84" Sep 18 07:01:16.745569 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:16.745519 2567 scope.go:117] "RemoveContainer" containerID="d3c966397f10bdefdb6513ccfeff8d55d3a69fd551fa62da98843aafa39f14c0" Sep 18 07:01:16.755823 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:16.755805 2567 scope.go:117] "RemoveContainer" containerID="0202dfab41b7ba25be88fdabe40fb70f500d7243bf23ba4a7468e9263dacdd84" Sep 18 07:01:16.756086 ip-10-0-142-104 kubenswrapper[2567]: E0918 07:01:16.756067 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0202dfab41b7ba25be88fdabe40fb70f500d7243bf23ba4a7468e9263dacdd84\": container with ID starting with 0202dfab41b7ba25be88fdabe40fb70f500d7243bf23ba4a7468e9263dacdd84 not found: ID does not exist" containerID="0202dfab41b7ba25be88fdabe40fb70f500d7243bf23ba4a7468e9263dacdd84" Sep 18 07:01:16.756143 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:16.756094 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0202dfab41b7ba25be88fdabe40fb70f500d7243bf23ba4a7468e9263dacdd84"} err="failed to get container status \"0202dfab41b7ba25be88fdabe40fb70f500d7243bf23ba4a7468e9263dacdd84\": rpc error: code = NotFound desc = could not find container \"0202dfab41b7ba25be88fdabe40fb70f500d7243bf23ba4a7468e9263dacdd84\": container with ID starting with 0202dfab41b7ba25be88fdabe40fb70f500d7243bf23ba4a7468e9263dacdd84 not found: ID does not exist" Sep 18 07:01:16.756143 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:16.756112 2567 scope.go:117] "RemoveContainer" containerID="d3c966397f10bdefdb6513ccfeff8d55d3a69fd551fa62da98843aafa39f14c0" Sep 18 07:01:16.756418 ip-10-0-142-104 kubenswrapper[2567]: E0918 07:01:16.756399 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3c966397f10bdefdb6513ccfeff8d55d3a69fd551fa62da98843aafa39f14c0\": container with ID starting with d3c966397f10bdefdb6513ccfeff8d55d3a69fd551fa62da98843aafa39f14c0 not found: ID does not exist" containerID="d3c966397f10bdefdb6513ccfeff8d55d3a69fd551fa62da98843aafa39f14c0" Sep 18 07:01:16.756470 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:16.756425 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3c966397f10bdefdb6513ccfeff8d55d3a69fd551fa62da98843aafa39f14c0"} err="failed to get container status \"d3c966397f10bdefdb6513ccfeff8d55d3a69fd551fa62da98843aafa39f14c0\": rpc error: code = NotFound desc = could not find container \"d3c966397f10bdefdb6513ccfeff8d55d3a69fd551fa62da98843aafa39f14c0\": container with ID starting with d3c966397f10bdefdb6513ccfeff8d55d3a69fd551fa62da98843aafa39f14c0 not found: ID does not exist" Sep 18 07:01:16.774437 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:16.774403 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nmlxq\" (UniqueName: \"kubernetes.io/projected/65ba9bd5-6fd8-489a-b30f-2481ca8be065-kube-api-access-nmlxq\") pod \"65ba9bd5-6fd8-489a-b30f-2481ca8be065\" (UID: \"65ba9bd5-6fd8-489a-b30f-2481ca8be065\") " Sep 18 07:01:16.774571 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:16.774480 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65ba9bd5-6fd8-489a-b30f-2481ca8be065-workload-certs\") pod \"65ba9bd5-6fd8-489a-b30f-2481ca8be065\" (UID: \"65ba9bd5-6fd8-489a-b30f-2481ca8be065\") " Sep 18 07:01:16.774571 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:16.774518 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65ba9bd5-6fd8-489a-b30f-2481ca8be065-istio-podinfo\") pod \"65ba9bd5-6fd8-489a-b30f-2481ca8be065\" (UID: \"65ba9bd5-6fd8-489a-b30f-2481ca8be065\") " Sep 18 07:01:16.774571 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:16.774557 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65ba9bd5-6fd8-489a-b30f-2481ca8be065-istio-envoy\") pod \"65ba9bd5-6fd8-489a-b30f-2481ca8be065\" (UID: \"65ba9bd5-6fd8-489a-b30f-2481ca8be065\") " Sep 18 07:01:16.774719 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:16.774584 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ba9bd5-6fd8-489a-b30f-2481ca8be065-workload-socket\") pod \"65ba9bd5-6fd8-489a-b30f-2481ca8be065\" (UID: \"65ba9bd5-6fd8-489a-b30f-2481ca8be065\") " Sep 18 07:01:16.774719 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:16.774622 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65ba9bd5-6fd8-489a-b30f-2481ca8be065-istio-token\") pod \"65ba9bd5-6fd8-489a-b30f-2481ca8be065\" (UID: \"65ba9bd5-6fd8-489a-b30f-2481ca8be065\") " Sep 18 07:01:16.774719 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:16.774678 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65ba9bd5-6fd8-489a-b30f-2481ca8be065-istio-data\") pod \"65ba9bd5-6fd8-489a-b30f-2481ca8be065\" (UID: \"65ba9bd5-6fd8-489a-b30f-2481ca8be065\") " Sep 18 07:01:16.774719 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:16.774710 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65ba9bd5-6fd8-489a-b30f-2481ca8be065-istiod-ca-cert\") pod \"65ba9bd5-6fd8-489a-b30f-2481ca8be065\" (UID: \"65ba9bd5-6fd8-489a-b30f-2481ca8be065\") " Sep 18 07:01:16.774927 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:16.774765 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ba9bd5-6fd8-489a-b30f-2481ca8be065-credential-socket\") pod \"65ba9bd5-6fd8-489a-b30f-2481ca8be065\" (UID: \"65ba9bd5-6fd8-489a-b30f-2481ca8be065\") " Sep 18 07:01:16.774927 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:16.774792 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65ba9bd5-6fd8-489a-b30f-2481ca8be065-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "65ba9bd5-6fd8-489a-b30f-2481ca8be065" (UID: "65ba9bd5-6fd8-489a-b30f-2481ca8be065"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 07:01:16.775040 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:16.774940 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65ba9bd5-6fd8-489a-b30f-2481ca8be065-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "65ba9bd5-6fd8-489a-b30f-2481ca8be065" (UID: "65ba9bd5-6fd8-489a-b30f-2481ca8be065"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 07:01:16.775040 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:16.774992 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65ba9bd5-6fd8-489a-b30f-2481ca8be065-istio-data" (OuterVolumeSpecName: "istio-data") pod "65ba9bd5-6fd8-489a-b30f-2481ca8be065" (UID: "65ba9bd5-6fd8-489a-b30f-2481ca8be065"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 07:01:16.775153 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:16.775121 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65ba9bd5-6fd8-489a-b30f-2481ca8be065-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "65ba9bd5-6fd8-489a-b30f-2481ca8be065" (UID: "65ba9bd5-6fd8-489a-b30f-2481ca8be065"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 07:01:16.775205 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:16.775186 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65ba9bd5-6fd8-489a-b30f-2481ca8be065-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "65ba9bd5-6fd8-489a-b30f-2481ca8be065" (UID: "65ba9bd5-6fd8-489a-b30f-2481ca8be065"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 07:01:16.775264 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:16.775201 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65ba9bd5-6fd8-489a-b30f-2481ca8be065-istio-data\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 07:01:16.775264 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:16.775229 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ba9bd5-6fd8-489a-b30f-2481ca8be065-credential-socket\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 07:01:16.775264 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:16.775243 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65ba9bd5-6fd8-489a-b30f-2481ca8be065-workload-certs\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 07:01:16.775264 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:16.775256 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ba9bd5-6fd8-489a-b30f-2481ca8be065-workload-socket\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 07:01:16.776819 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:16.776779 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65ba9bd5-6fd8-489a-b30f-2481ca8be065-kube-api-access-nmlxq" (OuterVolumeSpecName: "kube-api-access-nmlxq") pod "65ba9bd5-6fd8-489a-b30f-2481ca8be065" (UID: "65ba9bd5-6fd8-489a-b30f-2481ca8be065"). InnerVolumeSpecName "kube-api-access-nmlxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 07:01:16.776924 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:16.776850 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65ba9bd5-6fd8-489a-b30f-2481ca8be065-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "65ba9bd5-6fd8-489a-b30f-2481ca8be065" (UID: "65ba9bd5-6fd8-489a-b30f-2481ca8be065"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 07:01:16.777051 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:16.777023 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65ba9bd5-6fd8-489a-b30f-2481ca8be065-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "65ba9bd5-6fd8-489a-b30f-2481ca8be065" (UID: "65ba9bd5-6fd8-489a-b30f-2481ca8be065"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 07:01:16.777217 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:16.777198 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65ba9bd5-6fd8-489a-b30f-2481ca8be065-istio-token" (OuterVolumeSpecName: "istio-token") pod "65ba9bd5-6fd8-489a-b30f-2481ca8be065" (UID: "65ba9bd5-6fd8-489a-b30f-2481ca8be065"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 07:01:16.876543 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:16.876505 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65ba9bd5-6fd8-489a-b30f-2481ca8be065-istio-token\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 07:01:16.876543 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:16.876539 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65ba9bd5-6fd8-489a-b30f-2481ca8be065-istiod-ca-cert\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 07:01:16.876543 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:16.876549 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmlxq\" (UniqueName: \"kubernetes.io/projected/65ba9bd5-6fd8-489a-b30f-2481ca8be065-kube-api-access-nmlxq\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 07:01:16.876790 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:16.876559 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65ba9bd5-6fd8-489a-b30f-2481ca8be065-istio-podinfo\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 07:01:16.876790 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:16.876569 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65ba9bd5-6fd8-489a-b30f-2481ca8be065-istio-envoy\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 07:01:17.063053 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:17.062996 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp"] Sep 18 07:01:17.066439 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:17.066407 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-75ce90-openshift-default-854df5bff4-nrspp"] Sep 18 07:01:17.893099 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:17.893068 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a7bd58-hr9zc" Sep 18 07:01:17.988157 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:17.988118 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a10adc9-1460-481c-9662-99112f5fd7ed-maas-api-service-ca\") pod \"6a10adc9-1460-481c-9662-99112f5fd7ed\" (UID: \"6a10adc9-1460-481c-9662-99112f5fd7ed\") " Sep 18 07:01:17.988484 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:17.988461 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a10adc9-1460-481c-9662-99112f5fd7ed-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6a10adc9-1460-481c-9662-99112f5fd7ed" (UID: "6a10adc9-1460-481c-9662-99112f5fd7ed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 07:01:18.089374 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:18.089340 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a10adc9-1460-481c-9662-99112f5fd7ed-maas-api-service-ca\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 07:01:18.448550 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:18.448472 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65ba9bd5-6fd8-489a-b30f-2481ca8be065" path="/var/lib/kubelet/pods/65ba9bd5-6fd8-489a-b30f-2481ca8be065/volumes" Sep 18 07:01:18.752301 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:18.752199 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a7bd58-hr9zc" Sep 18 07:01:18.752301 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:18.752223 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a7bd58-hr9zc" event={"ID":"6a10adc9-1460-481c-9662-99112f5fd7ed","Type":"ContainerDied","Data":"0da941467d4a7a107cf585f66af7a74ddf6c786a4dafac88119c1aa4a065ac01"} Sep 18 07:01:18.752301 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:18.752275 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0da941467d4a7a107cf585f66af7a74ddf6c786a4dafac88119c1aa4a065ac01" Sep 18 07:01:21.009306 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:21.009274 2567 scope.go:117] "RemoveContainer" containerID="60b37becd7a6ea35b3cd07ccd807529911f25aaf63f5b7dd704f5ed68cef55a6" Sep 18 07:01:21.018638 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:21.018616 2567 scope.go:117] "RemoveContainer" containerID="4b9977a109b5f49fe314a8c0c62d379e672b5b5e74a7b9c278f19b867a82140d" Sep 18 07:01:30.333864 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:30.333825 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a7bd58-29828580-2zpbw"] Sep 18 07:01:30.336822 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:30.336793 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a7bd58-29828580-2zpbw"] Sep 18 07:01:30.359016 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:30.358978 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a7bd58-6999dbc775-qcqlf"] Sep 18 07:01:30.359316 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:30.359258 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-a7bd58-6999dbc775-qcqlf" podUID="98d614d7-ff85-4f64-b4a2-d85549dd05af" containerName="payload-processing" containerID="cri-o://f39934de0c56519be35d256757e854d61035356e9577120ffa02e03e356f117d" gracePeriod=30 Sep 18 07:01:30.363480 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:30.363458 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a7bd58-54468db6f8-4tdxf"] Sep 18 07:01:30.364644 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:30.364559 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a7bd58-54468db6f8-4tdxf" podUID="d78179e1-1ee6-4767-9496-8a1973cb590a" containerName="payload-pre-processing" containerID="cri-o://43670c11305976d4c1019639d32256419f4ff9ef83ee82cb128482c96a048f6c" gracePeriod=30 Sep 18 07:01:30.447475 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:30.447446 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20d096dd-8b74-45c6-a57e-2303eefa38dc" path="/var/lib/kubelet/pods/20d096dd-8b74-45c6-a57e-2303eefa38dc/volumes" Sep 18 07:01:30.512081 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:30.512044 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a7bd58-hr9zc"] Sep 18 07:01:30.515228 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:30.515197 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a7bd58-hr9zc"] Sep 18 07:01:30.653293 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:30.653264 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a7bd58-6999dbc775-qcqlf" Sep 18 07:01:30.656464 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:30.656443 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a7bd58-54468db6f8-4tdxf" Sep 18 07:01:30.816475 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:30.816438 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-trvc4\" (UniqueName: \"kubernetes.io/projected/d78179e1-1ee6-4767-9496-8a1973cb590a-kube-api-access-trvc4\") pod \"d78179e1-1ee6-4767-9496-8a1973cb590a\" (UID: \"d78179e1-1ee6-4767-9496-8a1973cb590a\") " Sep 18 07:01:30.816688 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:30.816535 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d78179e1-1ee6-4767-9496-8a1973cb590a-plugins-config-volume\") pod \"d78179e1-1ee6-4767-9496-8a1973cb590a\" (UID: \"d78179e1-1ee6-4767-9496-8a1973cb590a\") " Sep 18 07:01:30.816688 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:30.816567 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98d614d7-ff85-4f64-b4a2-d85549dd05af-plugins-config-volume\") pod \"98d614d7-ff85-4f64-b4a2-d85549dd05af\" (UID: \"98d614d7-ff85-4f64-b4a2-d85549dd05af\") " Sep 18 07:01:30.816688 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:30.816588 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-frc2c\" (UniqueName: \"kubernetes.io/projected/98d614d7-ff85-4f64-b4a2-d85549dd05af-kube-api-access-frc2c\") pod \"98d614d7-ff85-4f64-b4a2-d85549dd05af\" (UID: \"98d614d7-ff85-4f64-b4a2-d85549dd05af\") " Sep 18 07:01:30.816958 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:30.816935 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d78179e1-1ee6-4767-9496-8a1973cb590a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d78179e1-1ee6-4767-9496-8a1973cb590a" (UID: "d78179e1-1ee6-4767-9496-8a1973cb590a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 07:01:30.817023 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:30.816964 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98d614d7-ff85-4f64-b4a2-d85549dd05af-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "98d614d7-ff85-4f64-b4a2-d85549dd05af" (UID: "98d614d7-ff85-4f64-b4a2-d85549dd05af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 07:01:30.818797 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:30.818779 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d78179e1-1ee6-4767-9496-8a1973cb590a-kube-api-access-trvc4" (OuterVolumeSpecName: "kube-api-access-trvc4") pod "d78179e1-1ee6-4767-9496-8a1973cb590a" (UID: "d78179e1-1ee6-4767-9496-8a1973cb590a"). InnerVolumeSpecName "kube-api-access-trvc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 07:01:30.818882 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:30.818819 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98d614d7-ff85-4f64-b4a2-d85549dd05af-kube-api-access-frc2c" (OuterVolumeSpecName: "kube-api-access-frc2c") pod "98d614d7-ff85-4f64-b4a2-d85549dd05af" (UID: "98d614d7-ff85-4f64-b4a2-d85549dd05af"). InnerVolumeSpecName "kube-api-access-frc2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 07:01:30.829792 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:30.829762 2567 generic.go:356] "Generic (PLEG): container finished" podID="d78179e1-1ee6-4767-9496-8a1973cb590a" containerID="43670c11305976d4c1019639d32256419f4ff9ef83ee82cb128482c96a048f6c" exitCode=0 Sep 18 07:01:30.829959 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:30.829809 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a7bd58-54468db6f8-4tdxf" Sep 18 07:01:30.829959 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:30.829823 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a7bd58-54468db6f8-4tdxf" event={"ID":"d78179e1-1ee6-4767-9496-8a1973cb590a","Type":"ContainerDied","Data":"43670c11305976d4c1019639d32256419f4ff9ef83ee82cb128482c96a048f6c"} Sep 18 07:01:30.829959 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:30.829940 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a7bd58-54468db6f8-4tdxf" event={"ID":"d78179e1-1ee6-4767-9496-8a1973cb590a","Type":"ContainerDied","Data":"039174019fd8757586e1f98f5ed96589a05069e031a9f5638501b8c10dc9d5cc"} Sep 18 07:01:30.830118 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:30.829974 2567 scope.go:117] "RemoveContainer" containerID="43670c11305976d4c1019639d32256419f4ff9ef83ee82cb128482c96a048f6c" Sep 18 07:01:30.831461 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:30.831435 2567 generic.go:356] "Generic (PLEG): container finished" podID="98d614d7-ff85-4f64-b4a2-d85549dd05af" containerID="f39934de0c56519be35d256757e854d61035356e9577120ffa02e03e356f117d" exitCode=0 Sep 18 07:01:30.831565 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:30.831480 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a7bd58-6999dbc775-qcqlf" Sep 18 07:01:30.831565 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:30.831508 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a7bd58-6999dbc775-qcqlf" event={"ID":"98d614d7-ff85-4f64-b4a2-d85549dd05af","Type":"ContainerDied","Data":"f39934de0c56519be35d256757e854d61035356e9577120ffa02e03e356f117d"} Sep 18 07:01:30.831565 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:30.831549 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a7bd58-6999dbc775-qcqlf" event={"ID":"98d614d7-ff85-4f64-b4a2-d85549dd05af","Type":"ContainerDied","Data":"65fb443cc10aa723df6965f0ab88b238946a59b708bfd8a16b8531c8b1096787"} Sep 18 07:01:30.841779 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:30.841761 2567 scope.go:117] "RemoveContainer" containerID="43670c11305976d4c1019639d32256419f4ff9ef83ee82cb128482c96a048f6c" Sep 18 07:01:30.842070 ip-10-0-142-104 kubenswrapper[2567]: E0918 07:01:30.842052 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43670c11305976d4c1019639d32256419f4ff9ef83ee82cb128482c96a048f6c\": container with ID starting with 43670c11305976d4c1019639d32256419f4ff9ef83ee82cb128482c96a048f6c not found: ID does not exist" containerID="43670c11305976d4c1019639d32256419f4ff9ef83ee82cb128482c96a048f6c" Sep 18 07:01:30.842122 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:30.842078 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43670c11305976d4c1019639d32256419f4ff9ef83ee82cb128482c96a048f6c"} err="failed to get container status \"43670c11305976d4c1019639d32256419f4ff9ef83ee82cb128482c96a048f6c\": rpc error: code = NotFound desc = could not find container \"43670c11305976d4c1019639d32256419f4ff9ef83ee82cb128482c96a048f6c\": container with ID starting with 43670c11305976d4c1019639d32256419f4ff9ef83ee82cb128482c96a048f6c not found: ID does not exist" Sep 18 07:01:30.842122 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:30.842095 2567 scope.go:117] "RemoveContainer" containerID="f39934de0c56519be35d256757e854d61035356e9577120ffa02e03e356f117d" Sep 18 07:01:30.851177 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:30.851154 2567 scope.go:117] "RemoveContainer" containerID="f39934de0c56519be35d256757e854d61035356e9577120ffa02e03e356f117d" Sep 18 07:01:30.851482 ip-10-0-142-104 kubenswrapper[2567]: E0918 07:01:30.851463 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f39934de0c56519be35d256757e854d61035356e9577120ffa02e03e356f117d\": container with ID starting with f39934de0c56519be35d256757e854d61035356e9577120ffa02e03e356f117d not found: ID does not exist" containerID="f39934de0c56519be35d256757e854d61035356e9577120ffa02e03e356f117d" Sep 18 07:01:30.851596 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:30.851493 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f39934de0c56519be35d256757e854d61035356e9577120ffa02e03e356f117d"} err="failed to get container status \"f39934de0c56519be35d256757e854d61035356e9577120ffa02e03e356f117d\": rpc error: code = NotFound desc = could not find container \"f39934de0c56519be35d256757e854d61035356e9577120ffa02e03e356f117d\": container with ID starting with f39934de0c56519be35d256757e854d61035356e9577120ffa02e03e356f117d not found: ID does not exist" Sep 18 07:01:30.857924 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:30.857894 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a7bd58-6999dbc775-qcqlf"] Sep 18 07:01:30.863271 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:30.863250 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a7bd58-6999dbc775-qcqlf"] Sep 18 07:01:30.872798 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:30.872769 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a7bd58-54468db6f8-4tdxf"] Sep 18 07:01:30.876620 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:30.876549 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a7bd58-54468db6f8-4tdxf"] Sep 18 07:01:30.917835 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:30.917801 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trvc4\" (UniqueName: \"kubernetes.io/projected/d78179e1-1ee6-4767-9496-8a1973cb590a-kube-api-access-trvc4\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 07:01:30.917835 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:30.917833 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d78179e1-1ee6-4767-9496-8a1973cb590a-plugins-config-volume\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 07:01:30.918092 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:30.917849 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98d614d7-ff85-4f64-b4a2-d85549dd05af-plugins-config-volume\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 07:01:30.918092 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:30.917859 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frc2c\" (UniqueName: \"kubernetes.io/projected/98d614d7-ff85-4f64-b4a2-d85549dd05af-kube-api-access-frc2c\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 07:01:32.448444 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:32.448411 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a10adc9-1460-481c-9662-99112f5fd7ed" path="/var/lib/kubelet/pods/6a10adc9-1460-481c-9662-99112f5fd7ed/volumes" Sep 18 07:01:32.448973 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:32.448952 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98d614d7-ff85-4f64-b4a2-d85549dd05af" path="/var/lib/kubelet/pods/98d614d7-ff85-4f64-b4a2-d85549dd05af/volumes" Sep 18 07:01:32.449357 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:01:32.449337 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d78179e1-1ee6-4767-9496-8a1973cb590a" path="/var/lib/kubelet/pods/d78179e1-1ee6-4767-9496-8a1973cb590a/volumes" Sep 18 07:02:08.412797 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:08.412761 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk"] Sep 18 07:02:08.413436 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:08.413038 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk" podUID="fcd19467-e9c3-462b-b277-9cb88354fc88" containerName="istio-proxy" containerID="cri-o://3df974ca684cdbaa6aa368e682417fa04f19c98a91c9170477800d85b68d1e6d" gracePeriod=30 Sep 18 07:02:08.762197 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:08.762089 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1672e6-kfb7t"] Sep 18 07:02:08.762637 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:08.762617 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65ba9bd5-6fd8-489a-b30f-2481ca8be065" containerName="istio-proxy" Sep 18 07:02:08.762748 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:08.762640 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="65ba9bd5-6fd8-489a-b30f-2481ca8be065" containerName="istio-proxy" Sep 18 07:02:08.762748 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:08.762681 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65ba9bd5-6fd8-489a-b30f-2481ca8be065" containerName="istio-proxy" Sep 18 07:02:08.762748 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:08.762689 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="65ba9bd5-6fd8-489a-b30f-2481ca8be065" containerName="istio-proxy" Sep 18 07:02:08.762748 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:08.762701 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98d614d7-ff85-4f64-b4a2-d85549dd05af" containerName="payload-processing" Sep 18 07:02:08.762748 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:08.762711 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="98d614d7-ff85-4f64-b4a2-d85549dd05af" containerName="payload-processing" Sep 18 07:02:08.762748 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:08.762723 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d78179e1-1ee6-4767-9496-8a1973cb590a" containerName="payload-pre-processing" Sep 18 07:02:08.762748 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:08.762732 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d78179e1-1ee6-4767-9496-8a1973cb590a" containerName="payload-pre-processing" Sep 18 07:02:08.762748 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:08.762747 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a10adc9-1460-481c-9662-99112f5fd7ed" containerName="revoke-keys" Sep 18 07:02:08.763144 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:08.762756 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a10adc9-1460-481c-9662-99112f5fd7ed" containerName="revoke-keys" Sep 18 07:02:08.763144 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:08.762875 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d78179e1-1ee6-4767-9496-8a1973cb590a" containerName="payload-pre-processing" Sep 18 07:02:08.763144 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:08.762887 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="98d614d7-ff85-4f64-b4a2-d85549dd05af" containerName="payload-processing" Sep 18 07:02:08.763144 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:08.762904 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="65ba9bd5-6fd8-489a-b30f-2481ca8be065" containerName="istio-proxy" Sep 18 07:02:08.763144 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:08.762914 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a10adc9-1460-481c-9662-99112f5fd7ed" containerName="revoke-keys" Sep 18 07:02:08.766299 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:08.766278 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1672e6-kfb7t" Sep 18 07:02:08.769002 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:08.768975 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 18 07:02:08.769196 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:08.768975 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-88t7h\"" Sep 18 07:02:08.772035 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:08.772007 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1672e6-kfb7t"] Sep 18 07:02:08.886343 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:08.886300 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c06e85a-37bf-4083-97a1-441f397e4812-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1672e6-kfb7t\" (UID: \"1c06e85a-37bf-4083-97a1-441f397e4812\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1672e6-kfb7t" Sep 18 07:02:08.987945 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:08.987909 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c06e85a-37bf-4083-97a1-441f397e4812-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1672e6-kfb7t\" (UID: \"1c06e85a-37bf-4083-97a1-441f397e4812\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1672e6-kfb7t" Sep 18 07:02:08.988587 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:08.988564 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c06e85a-37bf-4083-97a1-441f397e4812-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1672e6-kfb7t\" (UID: \"1c06e85a-37bf-4083-97a1-441f397e4812\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1672e6-kfb7t" Sep 18 07:02:09.079929 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:09.079846 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1672e6-kfb7t" Sep 18 07:02:09.357267 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:09.357234 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1672e6-kfb7t"] Sep 18 07:02:09.358802 ip-10-0-142-104 kubenswrapper[2567]: W0918 07:02:09.358766 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c06e85a_37bf_4083_97a1_441f397e4812.slice/crio-007a06ab6aece508d9d246ed681f44720157566960b73795963d6aadc1e4c16c WatchSource:0}: Error finding container 007a06ab6aece508d9d246ed681f44720157566960b73795963d6aadc1e4c16c: Status 404 returned error can't find the container with id 007a06ab6aece508d9d246ed681f44720157566960b73795963d6aadc1e4c16c Sep 18 07:02:10.070847 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:10.070817 2567 generic.go:356] "Generic (PLEG): container finished" podID="1c06e85a-37bf-4083-97a1-441f397e4812" containerID="9d5493fc731346e4c93438d54117d7f9d92f44b88bf4bbfbaf92ccf00d345f01" exitCode=0 Sep 18 07:02:10.071187 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:10.070898 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1672e6-kfb7t" event={"ID":"1c06e85a-37bf-4083-97a1-441f397e4812","Type":"ContainerDied","Data":"9d5493fc731346e4c93438d54117d7f9d92f44b88bf4bbfbaf92ccf00d345f01"} Sep 18 07:02:10.071187 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:10.070948 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1672e6-kfb7t" event={"ID":"1c06e85a-37bf-4083-97a1-441f397e4812","Type":"ContainerStarted","Data":"007a06ab6aece508d9d246ed681f44720157566960b73795963d6aadc1e4c16c"} Sep 18 07:02:11.223548 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:11.223522 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1672e6-kfb7t" Sep 18 07:02:11.411411 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:11.411350 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c06e85a-37bf-4083-97a1-441f397e4812-maas-api-service-ca\") pod \"1c06e85a-37bf-4083-97a1-441f397e4812\" (UID: \"1c06e85a-37bf-4083-97a1-441f397e4812\") " Sep 18 07:02:11.411722 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:11.411697 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c06e85a-37bf-4083-97a1-441f397e4812-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1c06e85a-37bf-4083-97a1-441f397e4812" (UID: "1c06e85a-37bf-4083-97a1-441f397e4812"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 07:02:11.512421 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:11.512353 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c06e85a-37bf-4083-97a1-441f397e4812-maas-api-service-ca\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 07:02:12.085006 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:12.084967 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1672e6-kfb7t" Sep 18 07:02:12.085221 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:12.084970 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1672e6-kfb7t" event={"ID":"1c06e85a-37bf-4083-97a1-441f397e4812","Type":"ContainerDied","Data":"007a06ab6aece508d9d246ed681f44720157566960b73795963d6aadc1e4c16c"} Sep 18 07:02:12.085221 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:12.085122 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="007a06ab6aece508d9d246ed681f44720157566960b73795963d6aadc1e4c16c" Sep 18 07:02:13.673248 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:13.673220 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk" Sep 18 07:02:13.836149 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:13.836106 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcd19467-e9c3-462b-b277-9cb88354fc88-istio-envoy\") pod \"fcd19467-e9c3-462b-b277-9cb88354fc88\" (UID: \"fcd19467-e9c3-462b-b277-9cb88354fc88\") " Sep 18 07:02:13.836451 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:13.836206 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcd19467-e9c3-462b-b277-9cb88354fc88-istio-token\") pod \"fcd19467-e9c3-462b-b277-9cb88354fc88\" (UID: \"fcd19467-e9c3-462b-b277-9cb88354fc88\") " Sep 18 07:02:13.836451 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:13.836227 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcd19467-e9c3-462b-b277-9cb88354fc88-istio-data\") pod \"fcd19467-e9c3-462b-b277-9cb88354fc88\" (UID: \"fcd19467-e9c3-462b-b277-9cb88354fc88\") " Sep 18 07:02:13.836451 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:13.836244 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcd19467-e9c3-462b-b277-9cb88354fc88-credential-socket\") pod \"fcd19467-e9c3-462b-b277-9cb88354fc88\" (UID: \"fcd19467-e9c3-462b-b277-9cb88354fc88\") " Sep 18 07:02:13.836451 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:13.836263 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-szl8d\" (UniqueName: \"kubernetes.io/projected/fcd19467-e9c3-462b-b277-9cb88354fc88-kube-api-access-szl8d\") pod \"fcd19467-e9c3-462b-b277-9cb88354fc88\" (UID: \"fcd19467-e9c3-462b-b277-9cb88354fc88\") " Sep 18 07:02:13.836451 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:13.836281 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcd19467-e9c3-462b-b277-9cb88354fc88-istiod-ca-cert\") pod \"fcd19467-e9c3-462b-b277-9cb88354fc88\" (UID: \"fcd19467-e9c3-462b-b277-9cb88354fc88\") " Sep 18 07:02:13.836451 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:13.836304 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcd19467-e9c3-462b-b277-9cb88354fc88-workload-socket\") pod \"fcd19467-e9c3-462b-b277-9cb88354fc88\" (UID: \"fcd19467-e9c3-462b-b277-9cb88354fc88\") " Sep 18 07:02:13.836792 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:13.836583 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcd19467-e9c3-462b-b277-9cb88354fc88-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fcd19467-e9c3-462b-b277-9cb88354fc88" (UID: "fcd19467-e9c3-462b-b277-9cb88354fc88"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 07:02:13.836792 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:13.836596 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcd19467-e9c3-462b-b277-9cb88354fc88-istio-data" (OuterVolumeSpecName: "istio-data") pod "fcd19467-e9c3-462b-b277-9cb88354fc88" (UID: "fcd19467-e9c3-462b-b277-9cb88354fc88"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 07:02:13.836792 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:13.836612 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcd19467-e9c3-462b-b277-9cb88354fc88-workload-certs\") pod \"fcd19467-e9c3-462b-b277-9cb88354fc88\" (UID: \"fcd19467-e9c3-462b-b277-9cb88354fc88\") " Sep 18 07:02:13.836792 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:13.836721 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcd19467-e9c3-462b-b277-9cb88354fc88-istio-podinfo\") pod \"fcd19467-e9c3-462b-b277-9cb88354fc88\" (UID: \"fcd19467-e9c3-462b-b277-9cb88354fc88\") " Sep 18 07:02:13.836792 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:13.836769 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcd19467-e9c3-462b-b277-9cb88354fc88-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fcd19467-e9c3-462b-b277-9cb88354fc88" (UID: "fcd19467-e9c3-462b-b277-9cb88354fc88"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 07:02:13.837062 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:13.836812 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcd19467-e9c3-462b-b277-9cb88354fc88-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fcd19467-e9c3-462b-b277-9cb88354fc88" (UID: "fcd19467-e9c3-462b-b277-9cb88354fc88"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 07:02:13.837062 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:13.836843 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcd19467-e9c3-462b-b277-9cb88354fc88-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fcd19467-e9c3-462b-b277-9cb88354fc88" (UID: "fcd19467-e9c3-462b-b277-9cb88354fc88"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 07:02:13.837174 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:13.837095 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcd19467-e9c3-462b-b277-9cb88354fc88-istio-data\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 07:02:13.837174 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:13.837115 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcd19467-e9c3-462b-b277-9cb88354fc88-credential-socket\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 07:02:13.837174 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:13.837133 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcd19467-e9c3-462b-b277-9cb88354fc88-istiod-ca-cert\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 07:02:13.837174 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:13.837147 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcd19467-e9c3-462b-b277-9cb88354fc88-workload-socket\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 07:02:13.837174 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:13.837160 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcd19467-e9c3-462b-b277-9cb88354fc88-workload-certs\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 07:02:13.838653 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:13.838619 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcd19467-e9c3-462b-b277-9cb88354fc88-istio-token" (OuterVolumeSpecName: "istio-token") pod "fcd19467-e9c3-462b-b277-9cb88354fc88" (UID: "fcd19467-e9c3-462b-b277-9cb88354fc88"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 07:02:13.838765 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:13.838677 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcd19467-e9c3-462b-b277-9cb88354fc88-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fcd19467-e9c3-462b-b277-9cb88354fc88" (UID: "fcd19467-e9c3-462b-b277-9cb88354fc88"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 07:02:13.838765 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:13.838684 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcd19467-e9c3-462b-b277-9cb88354fc88-kube-api-access-szl8d" (OuterVolumeSpecName: "kube-api-access-szl8d") pod "fcd19467-e9c3-462b-b277-9cb88354fc88" (UID: "fcd19467-e9c3-462b-b277-9cb88354fc88"). InnerVolumeSpecName "kube-api-access-szl8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 07:02:13.838913 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:13.838895 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fcd19467-e9c3-462b-b277-9cb88354fc88-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fcd19467-e9c3-462b-b277-9cb88354fc88" (UID: "fcd19467-e9c3-462b-b277-9cb88354fc88"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 07:02:13.937908 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:13.937877 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcd19467-e9c3-462b-b277-9cb88354fc88-istio-token\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 07:02:13.937908 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:13.937907 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szl8d\" (UniqueName: \"kubernetes.io/projected/fcd19467-e9c3-462b-b277-9cb88354fc88-kube-api-access-szl8d\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 07:02:13.938132 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:13.937918 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcd19467-e9c3-462b-b277-9cb88354fc88-istio-podinfo\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 07:02:13.938132 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:13.937927 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcd19467-e9c3-462b-b277-9cb88354fc88-istio-envoy\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 07:02:14.099870 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:14.099775 2567 generic.go:356] "Generic (PLEG): container finished" podID="fcd19467-e9c3-462b-b277-9cb88354fc88" containerID="3df974ca684cdbaa6aa368e682417fa04f19c98a91c9170477800d85b68d1e6d" exitCode=0 Sep 18 07:02:14.099870 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:14.099828 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk" Sep 18 07:02:14.100114 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:14.099865 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk" event={"ID":"fcd19467-e9c3-462b-b277-9cb88354fc88","Type":"ContainerDied","Data":"3df974ca684cdbaa6aa368e682417fa04f19c98a91c9170477800d85b68d1e6d"} Sep 18 07:02:14.100114 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:14.099930 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk" event={"ID":"fcd19467-e9c3-462b-b277-9cb88354fc88","Type":"ContainerDied","Data":"a265667deaa9e0f7ca6c4bb9788fc57eeddbef0d0bf425c64fae2849cd5ea3f5"} Sep 18 07:02:14.100114 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:14.099963 2567 scope.go:117] "RemoveContainer" containerID="3df974ca684cdbaa6aa368e682417fa04f19c98a91c9170477800d85b68d1e6d" Sep 18 07:02:14.110570 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:14.110553 2567 scope.go:117] "RemoveContainer" containerID="2862ac6ba4054aff8f4de8c2d7d865b4955018f9f9d2a4770cfaeaf019c8991f" Sep 18 07:02:14.119820 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:14.119800 2567 scope.go:117] "RemoveContainer" containerID="3df974ca684cdbaa6aa368e682417fa04f19c98a91c9170477800d85b68d1e6d" Sep 18 07:02:14.120115 ip-10-0-142-104 kubenswrapper[2567]: E0918 07:02:14.120097 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3df974ca684cdbaa6aa368e682417fa04f19c98a91c9170477800d85b68d1e6d\": container with ID starting with 3df974ca684cdbaa6aa368e682417fa04f19c98a91c9170477800d85b68d1e6d not found: ID does not exist" containerID="3df974ca684cdbaa6aa368e682417fa04f19c98a91c9170477800d85b68d1e6d" Sep 18 07:02:14.120164 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:14.120123 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3df974ca684cdbaa6aa368e682417fa04f19c98a91c9170477800d85b68d1e6d"} err="failed to get container status \"3df974ca684cdbaa6aa368e682417fa04f19c98a91c9170477800d85b68d1e6d\": rpc error: code = NotFound desc = could not find container \"3df974ca684cdbaa6aa368e682417fa04f19c98a91c9170477800d85b68d1e6d\": container with ID starting with 3df974ca684cdbaa6aa368e682417fa04f19c98a91c9170477800d85b68d1e6d not found: ID does not exist" Sep 18 07:02:14.120164 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:14.120142 2567 scope.go:117] "RemoveContainer" containerID="2862ac6ba4054aff8f4de8c2d7d865b4955018f9f9d2a4770cfaeaf019c8991f" Sep 18 07:02:14.120472 ip-10-0-142-104 kubenswrapper[2567]: E0918 07:02:14.120455 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2862ac6ba4054aff8f4de8c2d7d865b4955018f9f9d2a4770cfaeaf019c8991f\": container with ID starting with 2862ac6ba4054aff8f4de8c2d7d865b4955018f9f9d2a4770cfaeaf019c8991f not found: ID does not exist" containerID="2862ac6ba4054aff8f4de8c2d7d865b4955018f9f9d2a4770cfaeaf019c8991f" Sep 18 07:02:14.120525 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:14.120480 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2862ac6ba4054aff8f4de8c2d7d865b4955018f9f9d2a4770cfaeaf019c8991f"} err="failed to get container status \"2862ac6ba4054aff8f4de8c2d7d865b4955018f9f9d2a4770cfaeaf019c8991f\": rpc error: code = NotFound desc = could not find container \"2862ac6ba4054aff8f4de8c2d7d865b4955018f9f9d2a4770cfaeaf019c8991f\": container with ID starting with 2862ac6ba4054aff8f4de8c2d7d865b4955018f9f9d2a4770cfaeaf019c8991f not found: ID does not exist" Sep 18 07:02:14.122326 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:14.122305 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk"] Sep 18 07:02:14.126207 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:14.126183 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-a7bd58-openshift-default-5658f87fb8-t8chk"] Sep 18 07:02:14.446677 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:14.446596 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcd19467-e9c3-462b-b277-9cb88354fc88" path="/var/lib/kubelet/pods/fcd19467-e9c3-462b-b277-9cb88354fc88/volumes" Sep 18 07:02:19.179225 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:19.179178 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1672e6-29828580-7sm2f"] Sep 18 07:02:19.181977 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:19.181949 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1672e6-29828580-7sm2f"] Sep 18 07:02:19.207180 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:19.206857 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1672e6-58bfdf58c7-7mvlb"] Sep 18 07:02:19.207540 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:19.207507 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-1672e6-58bfdf58c7-7mvlb" podUID="c1e71afc-e8f0-4bc8-b1b1-93d16f5c1641" containerName="payload-processing" containerID="cri-o://51cf4fb122ba43f0387cdf1518f44262ec542fa6a4e22f783e00b2cca131ddb8" gracePeriod=30 Sep 18 07:02:19.212621 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:19.212467 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1672e6-b445c6b9d-lsp7s"] Sep 18 07:02:19.212970 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:19.212789 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1672e6-b445c6b9d-lsp7s" podUID="45abc740-d22c-4fcc-af21-e75a01952cd6" containerName="payload-pre-processing" containerID="cri-o://3ba4878b7f617d653394a9e8c6bab5c11f60f8bc8ba5cb6b1fa2372255ff4725" gracePeriod=30 Sep 18 07:02:19.348225 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:19.348178 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1672e6-kfb7t"] Sep 18 07:02:19.350831 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:19.350799 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1672e6-kfb7t"] Sep 18 07:02:19.496937 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:19.496914 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1672e6-58bfdf58c7-7mvlb" Sep 18 07:02:19.499934 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:19.499914 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1672e6-b445c6b9d-lsp7s" Sep 18 07:02:19.581131 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:19.581092 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ptjd6\" (UniqueName: \"kubernetes.io/projected/c1e71afc-e8f0-4bc8-b1b1-93d16f5c1641-kube-api-access-ptjd6\") pod \"c1e71afc-e8f0-4bc8-b1b1-93d16f5c1641\" (UID: \"c1e71afc-e8f0-4bc8-b1b1-93d16f5c1641\") " Sep 18 07:02:19.581324 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:19.581154 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1e71afc-e8f0-4bc8-b1b1-93d16f5c1641-plugins-config-volume\") pod \"c1e71afc-e8f0-4bc8-b1b1-93d16f5c1641\" (UID: \"c1e71afc-e8f0-4bc8-b1b1-93d16f5c1641\") " Sep 18 07:02:19.581324 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:19.581180 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45abc740-d22c-4fcc-af21-e75a01952cd6-plugins-config-volume\") pod \"45abc740-d22c-4fcc-af21-e75a01952cd6\" (UID: \"45abc740-d22c-4fcc-af21-e75a01952cd6\") " Sep 18 07:02:19.581324 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:19.581209 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lzn6z\" (UniqueName: \"kubernetes.io/projected/45abc740-d22c-4fcc-af21-e75a01952cd6-kube-api-access-lzn6z\") pod \"45abc740-d22c-4fcc-af21-e75a01952cd6\" (UID: \"45abc740-d22c-4fcc-af21-e75a01952cd6\") " Sep 18 07:02:19.581626 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:19.581597 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45abc740-d22c-4fcc-af21-e75a01952cd6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "45abc740-d22c-4fcc-af21-e75a01952cd6" (UID: "45abc740-d22c-4fcc-af21-e75a01952cd6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 07:02:19.581626 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:19.581608 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1e71afc-e8f0-4bc8-b1b1-93d16f5c1641-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c1e71afc-e8f0-4bc8-b1b1-93d16f5c1641" (UID: "c1e71afc-e8f0-4bc8-b1b1-93d16f5c1641"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 07:02:19.583233 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:19.583213 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1e71afc-e8f0-4bc8-b1b1-93d16f5c1641-kube-api-access-ptjd6" (OuterVolumeSpecName: "kube-api-access-ptjd6") pod "c1e71afc-e8f0-4bc8-b1b1-93d16f5c1641" (UID: "c1e71afc-e8f0-4bc8-b1b1-93d16f5c1641"). InnerVolumeSpecName "kube-api-access-ptjd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 07:02:19.583308 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:19.583279 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45abc740-d22c-4fcc-af21-e75a01952cd6-kube-api-access-lzn6z" (OuterVolumeSpecName: "kube-api-access-lzn6z") pod "45abc740-d22c-4fcc-af21-e75a01952cd6" (UID: "45abc740-d22c-4fcc-af21-e75a01952cd6"). InnerVolumeSpecName "kube-api-access-lzn6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 07:02:19.682521 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:19.682487 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptjd6\" (UniqueName: \"kubernetes.io/projected/c1e71afc-e8f0-4bc8-b1b1-93d16f5c1641-kube-api-access-ptjd6\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 07:02:19.682521 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:19.682515 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1e71afc-e8f0-4bc8-b1b1-93d16f5c1641-plugins-config-volume\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 07:02:19.682521 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:19.682524 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45abc740-d22c-4fcc-af21-e75a01952cd6-plugins-config-volume\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 07:02:19.682752 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:19.682533 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzn6z\" (UniqueName: \"kubernetes.io/projected/45abc740-d22c-4fcc-af21-e75a01952cd6-kube-api-access-lzn6z\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 07:02:20.138586 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:20.138549 2567 generic.go:356] "Generic (PLEG): container finished" podID="c1e71afc-e8f0-4bc8-b1b1-93d16f5c1641" containerID="51cf4fb122ba43f0387cdf1518f44262ec542fa6a4e22f783e00b2cca131ddb8" exitCode=0 Sep 18 07:02:20.138797 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:20.138597 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1672e6-58bfdf58c7-7mvlb" Sep 18 07:02:20.138797 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:20.138634 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1672e6-58bfdf58c7-7mvlb" event={"ID":"c1e71afc-e8f0-4bc8-b1b1-93d16f5c1641","Type":"ContainerDied","Data":"51cf4fb122ba43f0387cdf1518f44262ec542fa6a4e22f783e00b2cca131ddb8"} Sep 18 07:02:20.138797 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:20.138691 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1672e6-58bfdf58c7-7mvlb" event={"ID":"c1e71afc-e8f0-4bc8-b1b1-93d16f5c1641","Type":"ContainerDied","Data":"08e2cbcf0bd7633bd2f22f4c142d167fda4262777d6a6b90d9e1986c98bcbbed"} Sep 18 07:02:20.138797 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:20.138716 2567 scope.go:117] "RemoveContainer" containerID="51cf4fb122ba43f0387cdf1518f44262ec542fa6a4e22f783e00b2cca131ddb8" Sep 18 07:02:20.140302 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:20.140277 2567 generic.go:356] "Generic (PLEG): container finished" podID="45abc740-d22c-4fcc-af21-e75a01952cd6" containerID="3ba4878b7f617d653394a9e8c6bab5c11f60f8bc8ba5cb6b1fa2372255ff4725" exitCode=0 Sep 18 07:02:20.140434 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:20.140325 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1672e6-b445c6b9d-lsp7s" Sep 18 07:02:20.140434 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:20.140388 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1672e6-b445c6b9d-lsp7s" event={"ID":"45abc740-d22c-4fcc-af21-e75a01952cd6","Type":"ContainerDied","Data":"3ba4878b7f617d653394a9e8c6bab5c11f60f8bc8ba5cb6b1fa2372255ff4725"} Sep 18 07:02:20.140528 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:20.140438 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1672e6-b445c6b9d-lsp7s" event={"ID":"45abc740-d22c-4fcc-af21-e75a01952cd6","Type":"ContainerDied","Data":"c4449366e827ccbca34b588cf62281ec52a1dc12bd7f228db752d502a8c6b0e4"} Sep 18 07:02:20.150026 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:20.150006 2567 scope.go:117] "RemoveContainer" containerID="51cf4fb122ba43f0387cdf1518f44262ec542fa6a4e22f783e00b2cca131ddb8" Sep 18 07:02:20.150406 ip-10-0-142-104 kubenswrapper[2567]: E0918 07:02:20.150328 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51cf4fb122ba43f0387cdf1518f44262ec542fa6a4e22f783e00b2cca131ddb8\": container with ID starting with 51cf4fb122ba43f0387cdf1518f44262ec542fa6a4e22f783e00b2cca131ddb8 not found: ID does not exist" containerID="51cf4fb122ba43f0387cdf1518f44262ec542fa6a4e22f783e00b2cca131ddb8" Sep 18 07:02:20.150406 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:20.150378 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51cf4fb122ba43f0387cdf1518f44262ec542fa6a4e22f783e00b2cca131ddb8"} err="failed to get container status \"51cf4fb122ba43f0387cdf1518f44262ec542fa6a4e22f783e00b2cca131ddb8\": rpc error: code = NotFound desc = could not find container \"51cf4fb122ba43f0387cdf1518f44262ec542fa6a4e22f783e00b2cca131ddb8\": container with ID starting with 51cf4fb122ba43f0387cdf1518f44262ec542fa6a4e22f783e00b2cca131ddb8 not found: ID does not exist" Sep 18 07:02:20.150406 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:20.150404 2567 scope.go:117] "RemoveContainer" containerID="3ba4878b7f617d653394a9e8c6bab5c11f60f8bc8ba5cb6b1fa2372255ff4725" Sep 18 07:02:20.160100 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:20.160080 2567 scope.go:117] "RemoveContainer" containerID="3ba4878b7f617d653394a9e8c6bab5c11f60f8bc8ba5cb6b1fa2372255ff4725" Sep 18 07:02:20.160437 ip-10-0-142-104 kubenswrapper[2567]: E0918 07:02:20.160418 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ba4878b7f617d653394a9e8c6bab5c11f60f8bc8ba5cb6b1fa2372255ff4725\": container with ID starting with 3ba4878b7f617d653394a9e8c6bab5c11f60f8bc8ba5cb6b1fa2372255ff4725 not found: ID does not exist" containerID="3ba4878b7f617d653394a9e8c6bab5c11f60f8bc8ba5cb6b1fa2372255ff4725" Sep 18 07:02:20.160492 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:20.160446 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ba4878b7f617d653394a9e8c6bab5c11f60f8bc8ba5cb6b1fa2372255ff4725"} err="failed to get container status \"3ba4878b7f617d653394a9e8c6bab5c11f60f8bc8ba5cb6b1fa2372255ff4725\": rpc error: code = NotFound desc = could not find container \"3ba4878b7f617d653394a9e8c6bab5c11f60f8bc8ba5cb6b1fa2372255ff4725\": container with ID starting with 3ba4878b7f617d653394a9e8c6bab5c11f60f8bc8ba5cb6b1fa2372255ff4725 not found: ID does not exist" Sep 18 07:02:20.161806 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:20.161786 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1672e6-58bfdf58c7-7mvlb"] Sep 18 07:02:20.165720 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:20.165699 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1672e6-58bfdf58c7-7mvlb"] Sep 18 07:02:20.175573 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:20.175535 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1672e6-b445c6b9d-lsp7s"] Sep 18 07:02:20.178495 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:20.178475 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1672e6-b445c6b9d-lsp7s"] Sep 18 07:02:20.453733 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:20.453651 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c06e85a-37bf-4083-97a1-441f397e4812" path="/var/lib/kubelet/pods/1c06e85a-37bf-4083-97a1-441f397e4812/volumes" Sep 18 07:02:20.454092 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:20.453953 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44beed82-dd93-41ef-afc5-206cb8053e84" path="/var/lib/kubelet/pods/44beed82-dd93-41ef-afc5-206cb8053e84/volumes" Sep 18 07:02:20.454232 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:20.454220 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45abc740-d22c-4fcc-af21-e75a01952cd6" path="/var/lib/kubelet/pods/45abc740-d22c-4fcc-af21-e75a01952cd6/volumes" Sep 18 07:02:20.454569 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:20.454556 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1e71afc-e8f0-4bc8-b1b1-93d16f5c1641" path="/var/lib/kubelet/pods/c1e71afc-e8f0-4bc8-b1b1-93d16f5c1641/volumes" Sep 18 07:02:20.640634 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:20.640600 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjr4h_eb059951-e263-40da-9155-4fd0cf242820/ovn-acl-logging/0.log" Sep 18 07:02:20.652318 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:20.652297 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjr4h_eb059951-e263-40da-9155-4fd0cf242820/ovn-acl-logging/0.log" Sep 18 07:02:25.929660 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:25.929620 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ec2077-tf7tw"] Sep 18 07:02:25.930044 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:25.930026 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45abc740-d22c-4fcc-af21-e75a01952cd6" containerName="payload-pre-processing" Sep 18 07:02:25.930044 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:25.930038 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="45abc740-d22c-4fcc-af21-e75a01952cd6" containerName="payload-pre-processing" Sep 18 07:02:25.930126 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:25.930048 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1e71afc-e8f0-4bc8-b1b1-93d16f5c1641" containerName="payload-processing" Sep 18 07:02:25.930126 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:25.930054 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1e71afc-e8f0-4bc8-b1b1-93d16f5c1641" containerName="payload-processing" Sep 18 07:02:25.930126 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:25.930062 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcd19467-e9c3-462b-b277-9cb88354fc88" containerName="istio-proxy" Sep 18 07:02:25.930126 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:25.930068 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcd19467-e9c3-462b-b277-9cb88354fc88" containerName="istio-proxy" Sep 18 07:02:25.930126 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:25.930082 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c06e85a-37bf-4083-97a1-441f397e4812" containerName="revoke-keys" Sep 18 07:02:25.930126 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:25.930087 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c06e85a-37bf-4083-97a1-441f397e4812" containerName="revoke-keys" Sep 18 07:02:25.930326 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:25.930145 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c06e85a-37bf-4083-97a1-441f397e4812" containerName="revoke-keys" Sep 18 07:02:25.930326 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:25.930152 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1e71afc-e8f0-4bc8-b1b1-93d16f5c1641" containerName="payload-processing" Sep 18 07:02:25.930326 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:25.930159 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcd19467-e9c3-462b-b277-9cb88354fc88" containerName="istio-proxy" Sep 18 07:02:25.930326 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:25.930165 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcd19467-e9c3-462b-b277-9cb88354fc88" containerName="istio-proxy" Sep 18 07:02:25.930326 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:25.930171 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="45abc740-d22c-4fcc-af21-e75a01952cd6" containerName="payload-pre-processing" Sep 18 07:02:25.930326 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:25.930177 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="65ba9bd5-6fd8-489a-b30f-2481ca8be065" containerName="istio-proxy" Sep 18 07:02:25.934985 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:25.934965 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ec2077-tf7tw" Sep 18 07:02:25.937874 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:25.937836 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-88t7h\"" Sep 18 07:02:25.937999 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:25.937840 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 18 07:02:25.944655 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:25.944629 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ec2077-tf7tw"] Sep 18 07:02:26.043425 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:26.043386 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58caaa58-7b5b-47b2-aace-3c908b3caad6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ec2077-tf7tw\" (UID: \"58caaa58-7b5b-47b2-aace-3c908b3caad6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ec2077-tf7tw" Sep 18 07:02:26.144495 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:26.144456 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58caaa58-7b5b-47b2-aace-3c908b3caad6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ec2077-tf7tw\" (UID: \"58caaa58-7b5b-47b2-aace-3c908b3caad6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ec2077-tf7tw" Sep 18 07:02:26.145094 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:26.145072 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58caaa58-7b5b-47b2-aace-3c908b3caad6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ec2077-tf7tw\" (UID: \"58caaa58-7b5b-47b2-aace-3c908b3caad6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ec2077-tf7tw" Sep 18 07:02:26.256878 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:26.256784 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ec2077-tf7tw" Sep 18 07:02:26.530269 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:26.530237 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ec2077-tf7tw"] Sep 18 07:02:26.531838 ip-10-0-142-104 kubenswrapper[2567]: W0918 07:02:26.531809 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58caaa58_7b5b_47b2_aace_3c908b3caad6.slice/crio-3f45dd4980bf575a5f25c4727b72c7024e393490d16668ae11fbae3ad0cc2af7 WatchSource:0}: Error finding container 3f45dd4980bf575a5f25c4727b72c7024e393490d16668ae11fbae3ad0cc2af7: Status 404 returned error can't find the container with id 3f45dd4980bf575a5f25c4727b72c7024e393490d16668ae11fbae3ad0cc2af7 Sep 18 07:02:27.183736 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:27.183708 2567 generic.go:356] "Generic (PLEG): container finished" podID="58caaa58-7b5b-47b2-aace-3c908b3caad6" containerID="59c7f7a5dd9e2a469c58c25d7230bc5005962aa25da5816d9e98dd5d8179fce0" exitCode=0 Sep 18 07:02:27.184092 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:27.183787 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ec2077-tf7tw" event={"ID":"58caaa58-7b5b-47b2-aace-3c908b3caad6","Type":"ContainerDied","Data":"59c7f7a5dd9e2a469c58c25d7230bc5005962aa25da5816d9e98dd5d8179fce0"} Sep 18 07:02:27.184092 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:27.183845 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ec2077-tf7tw" event={"ID":"58caaa58-7b5b-47b2-aace-3c908b3caad6","Type":"ContainerStarted","Data":"3f45dd4980bf575a5f25c4727b72c7024e393490d16668ae11fbae3ad0cc2af7"} Sep 18 07:02:28.324554 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:28.324515 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ec2077-tf7tw" Sep 18 07:02:28.363662 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:28.363632 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58caaa58-7b5b-47b2-aace-3c908b3caad6-maas-api-service-ca\") pod \"58caaa58-7b5b-47b2-aace-3c908b3caad6\" (UID: \"58caaa58-7b5b-47b2-aace-3c908b3caad6\") " Sep 18 07:02:28.364004 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:28.363983 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58caaa58-7b5b-47b2-aace-3c908b3caad6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "58caaa58-7b5b-47b2-aace-3c908b3caad6" (UID: "58caaa58-7b5b-47b2-aace-3c908b3caad6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 07:02:28.465193 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:28.465163 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58caaa58-7b5b-47b2-aace-3c908b3caad6-maas-api-service-ca\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 07:02:29.197450 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:29.197416 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ec2077-tf7tw" event={"ID":"58caaa58-7b5b-47b2-aace-3c908b3caad6","Type":"ContainerDied","Data":"3f45dd4980bf575a5f25c4727b72c7024e393490d16668ae11fbae3ad0cc2af7"} Sep 18 07:02:29.197637 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:29.197472 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f45dd4980bf575a5f25c4727b72c7024e393490d16668ae11fbae3ad0cc2af7" Sep 18 07:02:29.197637 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:29.197431 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ec2077-tf7tw" Sep 18 07:02:36.724064 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:36.723948 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ec2077-tf7tw"] Sep 18 07:02:36.725701 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:36.725675 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ec2077-tf7tw"] Sep 18 07:02:38.447489 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:38.447455 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58caaa58-7b5b-47b2-aace-3c908b3caad6" path="/var/lib/kubelet/pods/58caaa58-7b5b-47b2-aace-3c908b3caad6/volumes" Sep 18 07:02:43.454322 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:43.454286 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8fdd5b-lgllx"] Sep 18 07:02:43.454945 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:43.454710 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58caaa58-7b5b-47b2-aace-3c908b3caad6" containerName="revoke-keys" Sep 18 07:02:43.454945 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:43.454723 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="58caaa58-7b5b-47b2-aace-3c908b3caad6" containerName="revoke-keys" Sep 18 07:02:43.454945 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:43.454737 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcd19467-e9c3-462b-b277-9cb88354fc88" containerName="istio-proxy" Sep 18 07:02:43.454945 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:43.454742 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcd19467-e9c3-462b-b277-9cb88354fc88" containerName="istio-proxy" Sep 18 07:02:43.454945 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:43.454811 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="58caaa58-7b5b-47b2-aace-3c908b3caad6" containerName="revoke-keys" Sep 18 07:02:43.459285 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:43.459267 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8fdd5b-lgllx" Sep 18 07:02:43.462390 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:43.462341 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-88t7h\"" Sep 18 07:02:43.462531 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:43.462346 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 18 07:02:43.465282 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:43.465251 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8fdd5b-lgllx"] Sep 18 07:02:43.613024 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:43.612986 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b16650c-8144-4009-a4b1-33097f54cf1e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-8fdd5b-lgllx\" (UID: \"9b16650c-8144-4009-a4b1-33097f54cf1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8fdd5b-lgllx" Sep 18 07:02:43.714013 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:43.713916 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b16650c-8144-4009-a4b1-33097f54cf1e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-8fdd5b-lgllx\" (UID: \"9b16650c-8144-4009-a4b1-33097f54cf1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8fdd5b-lgllx" Sep 18 07:02:43.714663 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:43.714640 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b16650c-8144-4009-a4b1-33097f54cf1e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-8fdd5b-lgllx\" (UID: \"9b16650c-8144-4009-a4b1-33097f54cf1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8fdd5b-lgllx" Sep 18 07:02:43.772189 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:43.772142 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8fdd5b-lgllx" Sep 18 07:02:44.051055 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:44.051027 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8fdd5b-lgllx"] Sep 18 07:02:44.052757 ip-10-0-142-104 kubenswrapper[2567]: W0918 07:02:44.052724 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b16650c_8144_4009_a4b1_33097f54cf1e.slice/crio-2bcba952a7810d52ddf37d904ccd078067c32f9793a57f44419f5a9ace85c690 WatchSource:0}: Error finding container 2bcba952a7810d52ddf37d904ccd078067c32f9793a57f44419f5a9ace85c690: Status 404 returned error can't find the container with id 2bcba952a7810d52ddf37d904ccd078067c32f9793a57f44419f5a9ace85c690 Sep 18 07:02:44.297202 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:44.297167 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8fdd5b-lgllx" event={"ID":"9b16650c-8144-4009-a4b1-33097f54cf1e","Type":"ContainerStarted","Data":"2bcba952a7810d52ddf37d904ccd078067c32f9793a57f44419f5a9ace85c690"} Sep 18 07:02:45.305827 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:45.305786 2567 generic.go:356] "Generic (PLEG): container finished" podID="9b16650c-8144-4009-a4b1-33097f54cf1e" containerID="09d5566434341e917b3f707e06f9219c7716ab4cc87631082af225ee3e27971b" exitCode=0 Sep 18 07:02:45.306217 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:45.305890 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8fdd5b-lgllx" event={"ID":"9b16650c-8144-4009-a4b1-33097f54cf1e","Type":"ContainerDied","Data":"09d5566434341e917b3f707e06f9219c7716ab4cc87631082af225ee3e27971b"} Sep 18 07:02:46.448970 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:46.448945 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8fdd5b-lgllx" Sep 18 07:02:46.541290 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:46.541251 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b16650c-8144-4009-a4b1-33097f54cf1e-maas-api-service-ca\") pod \"9b16650c-8144-4009-a4b1-33097f54cf1e\" (UID: \"9b16650c-8144-4009-a4b1-33097f54cf1e\") " Sep 18 07:02:46.541670 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:46.541641 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b16650c-8144-4009-a4b1-33097f54cf1e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9b16650c-8144-4009-a4b1-33097f54cf1e" (UID: "9b16650c-8144-4009-a4b1-33097f54cf1e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 07:02:46.643130 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:46.643093 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b16650c-8144-4009-a4b1-33097f54cf1e-maas-api-service-ca\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 07:02:47.319517 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:47.319484 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8fdd5b-lgllx" Sep 18 07:02:47.319701 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:47.319528 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8fdd5b-lgllx" event={"ID":"9b16650c-8144-4009-a4b1-33097f54cf1e","Type":"ContainerDied","Data":"2bcba952a7810d52ddf37d904ccd078067c32f9793a57f44419f5a9ace85c690"} Sep 18 07:02:47.319701 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:47.319578 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bcba952a7810d52ddf37d904ccd078067c32f9793a57f44419f5a9ace85c690" Sep 18 07:02:54.048289 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:54.048245 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-8fdd5b-f766cfdc5-j9jrv"] Sep 18 07:02:54.049475 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:54.049438 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8fdd5b-f766cfdc5-j9jrv" podUID="e7041e5b-1691-422c-8f05-e12161264694" containerName="payload-processing" containerID="cri-o://431eb499dc6566285cb4b9e6e9f6933130e91cee78515d6e21ec27201b2ec2be" gracePeriod=30 Sep 18 07:02:54.058895 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:54.058869 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8fdd5b-76c7c7b74-z97ww"] Sep 18 07:02:54.059218 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:54.059188 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8fdd5b-76c7c7b74-z97ww" podUID="fc417726-1f8b-4510-82e7-882bf6c02d37" containerName="payload-pre-processing" containerID="cri-o://4e214d3ab42bf319dd1508c1391ad2876e2c12bca6f2cacdfdfec5198f8eab23" gracePeriod=30 Sep 18 07:02:54.192973 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:54.192930 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8fdd5b-lgllx"] Sep 18 07:02:54.195521 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:54.195493 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8fdd5b-lgllx"] Sep 18 07:02:54.321252 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:54.321226 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8fdd5b-f766cfdc5-j9jrv" Sep 18 07:02:54.339172 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:54.338882 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8fdd5b-76c7c7b74-z97ww" Sep 18 07:02:54.363403 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:54.363320 2567 generic.go:356] "Generic (PLEG): container finished" podID="fc417726-1f8b-4510-82e7-882bf6c02d37" containerID="4e214d3ab42bf319dd1508c1391ad2876e2c12bca6f2cacdfdfec5198f8eab23" exitCode=0 Sep 18 07:02:54.363726 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:54.363484 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8fdd5b-76c7c7b74-z97ww" event={"ID":"fc417726-1f8b-4510-82e7-882bf6c02d37","Type":"ContainerDied","Data":"4e214d3ab42bf319dd1508c1391ad2876e2c12bca6f2cacdfdfec5198f8eab23"} Sep 18 07:02:54.363848 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:54.363779 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8fdd5b-76c7c7b74-z97ww" event={"ID":"fc417726-1f8b-4510-82e7-882bf6c02d37","Type":"ContainerDied","Data":"8077d91d3f3bc77ecea0956b6ee57bb3c4d496ef6815badf2a15053aa6a48620"} Sep 18 07:02:54.363848 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:54.363818 2567 scope.go:117] "RemoveContainer" containerID="4e214d3ab42bf319dd1508c1391ad2876e2c12bca6f2cacdfdfec5198f8eab23" Sep 18 07:02:54.363972 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:54.363607 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8fdd5b-76c7c7b74-z97ww" Sep 18 07:02:54.365806 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:54.365783 2567 generic.go:356] "Generic (PLEG): container finished" podID="e7041e5b-1691-422c-8f05-e12161264694" containerID="431eb499dc6566285cb4b9e6e9f6933130e91cee78515d6e21ec27201b2ec2be" exitCode=0 Sep 18 07:02:54.365935 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:54.365824 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8fdd5b-f766cfdc5-j9jrv" Sep 18 07:02:54.365935 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:54.365836 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8fdd5b-f766cfdc5-j9jrv" event={"ID":"e7041e5b-1691-422c-8f05-e12161264694","Type":"ContainerDied","Data":"431eb499dc6566285cb4b9e6e9f6933130e91cee78515d6e21ec27201b2ec2be"} Sep 18 07:02:54.365935 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:54.365882 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8fdd5b-f766cfdc5-j9jrv" event={"ID":"e7041e5b-1691-422c-8f05-e12161264694","Type":"ContainerDied","Data":"6ccae1b97c08dfe115dbec06eb8fcb03f8336b69b2b62ecfc3931f4168261ca0"} Sep 18 07:02:54.379189 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:54.379165 2567 scope.go:117] "RemoveContainer" containerID="4e214d3ab42bf319dd1508c1391ad2876e2c12bca6f2cacdfdfec5198f8eab23" Sep 18 07:02:54.379632 ip-10-0-142-104 kubenswrapper[2567]: E0918 07:02:54.379600 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e214d3ab42bf319dd1508c1391ad2876e2c12bca6f2cacdfdfec5198f8eab23\": container with ID starting with 4e214d3ab42bf319dd1508c1391ad2876e2c12bca6f2cacdfdfec5198f8eab23 not found: ID does not exist" containerID="4e214d3ab42bf319dd1508c1391ad2876e2c12bca6f2cacdfdfec5198f8eab23" Sep 18 07:02:54.379762 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:54.379636 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e214d3ab42bf319dd1508c1391ad2876e2c12bca6f2cacdfdfec5198f8eab23"} err="failed to get container status \"4e214d3ab42bf319dd1508c1391ad2876e2c12bca6f2cacdfdfec5198f8eab23\": rpc error: code = NotFound desc = could not find container \"4e214d3ab42bf319dd1508c1391ad2876e2c12bca6f2cacdfdfec5198f8eab23\": container with ID starting with 4e214d3ab42bf319dd1508c1391ad2876e2c12bca6f2cacdfdfec5198f8eab23 not found: ID does not exist" Sep 18 07:02:54.379762 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:54.379659 2567 scope.go:117] "RemoveContainer" containerID="431eb499dc6566285cb4b9e6e9f6933130e91cee78515d6e21ec27201b2ec2be" Sep 18 07:02:54.388931 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:54.388912 2567 scope.go:117] "RemoveContainer" containerID="431eb499dc6566285cb4b9e6e9f6933130e91cee78515d6e21ec27201b2ec2be" Sep 18 07:02:54.389321 ip-10-0-142-104 kubenswrapper[2567]: E0918 07:02:54.389300 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"431eb499dc6566285cb4b9e6e9f6933130e91cee78515d6e21ec27201b2ec2be\": container with ID starting with 431eb499dc6566285cb4b9e6e9f6933130e91cee78515d6e21ec27201b2ec2be not found: ID does not exist" containerID="431eb499dc6566285cb4b9e6e9f6933130e91cee78515d6e21ec27201b2ec2be" Sep 18 07:02:54.389434 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:54.389332 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"431eb499dc6566285cb4b9e6e9f6933130e91cee78515d6e21ec27201b2ec2be"} err="failed to get container status \"431eb499dc6566285cb4b9e6e9f6933130e91cee78515d6e21ec27201b2ec2be\": rpc error: code = NotFound desc = could not find container \"431eb499dc6566285cb4b9e6e9f6933130e91cee78515d6e21ec27201b2ec2be\": container with ID starting with 431eb499dc6566285cb4b9e6e9f6933130e91cee78515d6e21ec27201b2ec2be not found: ID does not exist" Sep 18 07:02:54.424703 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:54.424670 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6trz\" (UniqueName: \"kubernetes.io/projected/e7041e5b-1691-422c-8f05-e12161264694-kube-api-access-m6trz\") pod \"e7041e5b-1691-422c-8f05-e12161264694\" (UID: \"e7041e5b-1691-422c-8f05-e12161264694\") " Sep 18 07:02:54.424873 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:54.424778 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7041e5b-1691-422c-8f05-e12161264694-plugins-config-volume\") pod \"e7041e5b-1691-422c-8f05-e12161264694\" (UID: \"e7041e5b-1691-422c-8f05-e12161264694\") " Sep 18 07:02:54.425131 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:54.425109 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7041e5b-1691-422c-8f05-e12161264694-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e7041e5b-1691-422c-8f05-e12161264694" (UID: "e7041e5b-1691-422c-8f05-e12161264694"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 07:02:54.426836 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:54.426814 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7041e5b-1691-422c-8f05-e12161264694-kube-api-access-m6trz" (OuterVolumeSpecName: "kube-api-access-m6trz") pod "e7041e5b-1691-422c-8f05-e12161264694" (UID: "e7041e5b-1691-422c-8f05-e12161264694"). InnerVolumeSpecName "kube-api-access-m6trz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 07:02:54.446048 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:54.446020 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b16650c-8144-4009-a4b1-33097f54cf1e" path="/var/lib/kubelet/pods/9b16650c-8144-4009-a4b1-33097f54cf1e/volumes" Sep 18 07:02:54.526001 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:54.525961 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc417726-1f8b-4510-82e7-882bf6c02d37-plugins-config-volume\") pod \"fc417726-1f8b-4510-82e7-882bf6c02d37\" (UID: \"fc417726-1f8b-4510-82e7-882bf6c02d37\") " Sep 18 07:02:54.526184 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:54.526018 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-59x4n\" (UniqueName: \"kubernetes.io/projected/fc417726-1f8b-4510-82e7-882bf6c02d37-kube-api-access-59x4n\") pod \"fc417726-1f8b-4510-82e7-882bf6c02d37\" (UID: \"fc417726-1f8b-4510-82e7-882bf6c02d37\") " Sep 18 07:02:54.526250 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:54.526227 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6trz\" (UniqueName: \"kubernetes.io/projected/e7041e5b-1691-422c-8f05-e12161264694-kube-api-access-m6trz\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 07:02:54.526250 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:54.526242 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7041e5b-1691-422c-8f05-e12161264694-plugins-config-volume\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 07:02:54.526357 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:54.526331 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc417726-1f8b-4510-82e7-882bf6c02d37-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fc417726-1f8b-4510-82e7-882bf6c02d37" (UID: "fc417726-1f8b-4510-82e7-882bf6c02d37"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 07:02:54.528162 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:54.528142 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc417726-1f8b-4510-82e7-882bf6c02d37-kube-api-access-59x4n" (OuterVolumeSpecName: "kube-api-access-59x4n") pod "fc417726-1f8b-4510-82e7-882bf6c02d37" (UID: "fc417726-1f8b-4510-82e7-882bf6c02d37"). InnerVolumeSpecName "kube-api-access-59x4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 07:02:54.627494 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:54.627457 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc417726-1f8b-4510-82e7-882bf6c02d37-plugins-config-volume\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 07:02:54.627494 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:54.627487 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59x4n\" (UniqueName: \"kubernetes.io/projected/fc417726-1f8b-4510-82e7-882bf6c02d37-kube-api-access-59x4n\") on node \"ip-10-0-142-104.ec2.internal\" DevicePath \"\"" Sep 18 07:02:54.683058 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:54.683025 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-8fdd5b-f766cfdc5-j9jrv"] Sep 18 07:02:54.687125 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:54.687098 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-8fdd5b-f766cfdc5-j9jrv"] Sep 18 07:02:54.697327 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:54.697298 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8fdd5b-76c7c7b74-z97ww"] Sep 18 07:02:54.702430 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:54.702402 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8fdd5b-76c7c7b74-z97ww"] Sep 18 07:02:56.446222 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:56.446186 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7041e5b-1691-422c-8f05-e12161264694" path="/var/lib/kubelet/pods/e7041e5b-1691-422c-8f05-e12161264694/volumes" Sep 18 07:02:56.446631 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:02:56.446558 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc417726-1f8b-4510-82e7-882bf6c02d37" path="/var/lib/kubelet/pods/fc417726-1f8b-4510-82e7-882bf6c02d37/volumes" Sep 18 07:06:21.127258 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:06:21.127212 2567 scope.go:117] "RemoveContainer" containerID="3d137e2bc5c86db8047c93a05b9c7686fc2b1a3cb2b8a8ce1e18865461a5e5ac" Sep 18 07:06:21.136580 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:06:21.136558 2567 scope.go:117] "RemoveContainer" containerID="a3dd1cb61de9b02718a2f28f94522a4917e6638305af15c408715141beb9c05f" Sep 18 07:07:20.686172 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:07:20.686135 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjr4h_eb059951-e263-40da-9155-4fd0cf242820/ovn-acl-logging/0.log" Sep 18 07:07:20.694296 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:07:20.694269 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjr4h_eb059951-e263-40da-9155-4fd0cf242820/ovn-acl-logging/0.log" Sep 18 07:07:21.162736 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:07:21.162705 2567 scope.go:117] "RemoveContainer" containerID="4a4923753ba3117596d1991269a2b18a38afb712c7dad5dbeabc5226932417b2" Sep 18 07:07:21.171722 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:07:21.171702 2567 scope.go:117] "RemoveContainer" containerID="2cb7b5f147b09a8485c4e995f188bac7215ab7104cecc936063f04b041d617d7" Sep 18 07:08:21.200530 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:08:21.200492 2567 scope.go:117] "RemoveContainer" containerID="9d5493fc731346e4c93438d54117d7f9d92f44b88bf4bbfbaf92ccf00d345f01" Sep 18 07:09:21.227135 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:21.227100 2567 scope.go:117] "RemoveContainer" containerID="59c7f7a5dd9e2a469c58c25d7230bc5005962aa25da5816d9e98dd5d8179fce0" Sep 18 07:09:21.236241 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:21.236218 2567 scope.go:117] "RemoveContainer" containerID="09d5566434341e917b3f707e06f9219c7716ab4cc87631082af225ee3e27971b" Sep 18 07:09:25.391381 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:25.391332 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-qgwmc_61e31de6-574c-43c5-a839-9dd1d9067587/manager/0.log" Sep 18 07:09:25.635615 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:25.635586 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6fd575c64f-7mcbn_44103c01-51b0-4d76-95e5-cb3b273f78ab/copy-manifests/0.log" Sep 18 07:09:25.651191 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:25.651113 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6fd575c64f-7mcbn_44103c01-51b0-4d76-95e5-cb3b273f78ab/manager/0.log" Sep 18 07:09:26.384968 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:26.384935 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29828565-g2dqc_2c07b0b7-0e00-44e1-b6f2-9b50968d0d8b/cleanup/0.log" Sep 18 07:09:26.503733 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:26.503703 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29828580-5krzm_40763e50-0e95-45b6-8df8-184fec55f673/cleanup/0.log" Sep 18 07:09:28.594352 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:28.594318 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg_2a4b02c0-6cda-40d5-a37a-05f0ba6ea4b4/istio-proxy/0.log" Sep 18 07:09:28.722161 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:28.722128 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-rzl5d_87337e45-0211-4fb0-bf69-7b19c2934525/discovery/0.log" Sep 18 07:09:29.335025 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:29.334993 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-jbzm5_96a61b61-7de1-428e-9fb6-99d5534fb7db/payload-processing/0.log" Sep 18 07:09:29.456280 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:29.456249 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-75b6d7b44b-fvj8r_e16d183f-ea45-4a4c-94be-f078321c3c79/router/0.log" Sep 18 07:09:29.842075 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:29.842045 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-hdknc_d43a82f2-a138-4b8e-917f-bf0b2ab4aa5c/main/0.log" Sep 18 07:09:30.230076 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:30.230001 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-2j9qq_7d2b5190-5648-44e7-b595-8e6346a0343d/main/0.log" Sep 18 07:09:39.678127 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:39.678098 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2jzcv_e57e7a14-2ca6-4f41-875f-822739fa1c43/global-pull-secret-syncer/0.log" Sep 18 07:09:40.066734 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:40.066648 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-sxrxx_631676f2-32af-4b0d-834b-41c7c59db5ba/konnectivity-agent/0.log" Sep 18 07:09:40.258076 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:40.258044 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-104.ec2.internal_f23d030455f844c3f1c0ef497274c121/haproxy/0.log" Sep 18 07:09:45.994585 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:45.994551 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-847b8bd8d7-tqkh6_c26181ee-5e97-4934-a7da-71a24e303856/cluster-monitoring-operator/0.log" Sep 18 07:09:46.032259 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:46.032228 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-4rqjv_3b9180cd-aece-4e1a-bd4f-781d7dd5ebf9/kube-state-metrics/0.log" Sep 18 07:09:46.066663 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:46.066637 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-4rqjv_3b9180cd-aece-4e1a-bd4f-781d7dd5ebf9/kube-rbac-proxy-main/0.log" Sep 18 07:09:46.095222 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:46.095197 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-4rqjv_3b9180cd-aece-4e1a-bd4f-781d7dd5ebf9/kube-rbac-proxy-self/0.log" Sep 18 07:09:46.128155 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:46.128126 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6df5dcb58c-gdgth_30956168-2c46-4c18-bd2a-d341ca614112/metrics-server/0.log" Sep 18 07:09:46.175679 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:46.175652 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5984459ff8-tb24m_eff3e28f-3d72-42c6-a14c-3080b1dacf59/monitoring-plugin/0.log" Sep 18 07:09:46.375434 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:46.375404 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nzscf_b642bccf-1547-424f-95ea-6de9749953a1/node-exporter/0.log" Sep 18 07:09:46.408526 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:46.408496 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nzscf_b642bccf-1547-424f-95ea-6de9749953a1/kube-rbac-proxy/0.log" Sep 18 07:09:46.436574 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:46.436549 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nzscf_b642bccf-1547-424f-95ea-6de9749953a1/init-textfile/0.log" Sep 18 07:09:46.567688 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:46.567658 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-24gd5_3fb98b73-fc76-4fab-928c-8c60bb29354c/kube-rbac-proxy-main/0.log" Sep 18 07:09:46.595616 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:46.595593 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-24gd5_3fb98b73-fc76-4fab-928c-8c60bb29354c/kube-rbac-proxy-self/0.log" Sep 18 07:09:46.626252 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:46.626179 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-24gd5_3fb98b73-fc76-4fab-928c-8c60bb29354c/openshift-state-metrics/0.log" Sep 18 07:09:47.289711 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:47.289686 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67d649cbb7-jbfsd_d5c0016b-5ad1-46f0-9460-ae6207e9d171/thanos-query/0.log" Sep 18 07:09:47.339589 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:47.339557 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67d649cbb7-jbfsd_d5c0016b-5ad1-46f0-9460-ae6207e9d171/kube-rbac-proxy-web/0.log" Sep 18 07:09:47.388989 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:47.388961 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67d649cbb7-jbfsd_d5c0016b-5ad1-46f0-9460-ae6207e9d171/kube-rbac-proxy/0.log" Sep 18 07:09:47.423845 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:47.423816 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67d649cbb7-jbfsd_d5c0016b-5ad1-46f0-9460-ae6207e9d171/prom-label-proxy/0.log" Sep 18 07:09:47.462894 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:47.462868 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67d649cbb7-jbfsd_d5c0016b-5ad1-46f0-9460-ae6207e9d171/kube-rbac-proxy-rules/0.log" Sep 18 07:09:47.499439 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:47.499411 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67d649cbb7-jbfsd_d5c0016b-5ad1-46f0-9460-ae6207e9d171/kube-rbac-proxy-metrics/0.log" Sep 18 07:09:49.691324 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:49.691283 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-87c5n/perf-node-gather-daemonset-ntpcl"] Sep 18 07:09:49.692147 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:49.691825 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7041e5b-1691-422c-8f05-e12161264694" containerName="payload-processing" Sep 18 07:09:49.692147 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:49.691842 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7041e5b-1691-422c-8f05-e12161264694" containerName="payload-processing" Sep 18 07:09:49.692147 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:49.691869 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b16650c-8144-4009-a4b1-33097f54cf1e" containerName="revoke-keys" Sep 18 07:09:49.692147 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:49.691875 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b16650c-8144-4009-a4b1-33097f54cf1e" containerName="revoke-keys" Sep 18 07:09:49.692147 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:49.691884 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc417726-1f8b-4510-82e7-882bf6c02d37" containerName="payload-pre-processing" Sep 18 07:09:49.692147 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:49.691892 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc417726-1f8b-4510-82e7-882bf6c02d37" containerName="payload-pre-processing" Sep 18 07:09:49.692147 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:49.691953 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b16650c-8144-4009-a4b1-33097f54cf1e" containerName="revoke-keys" Sep 18 07:09:49.692147 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:49.691962 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7041e5b-1691-422c-8f05-e12161264694" containerName="payload-processing" Sep 18 07:09:49.692147 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:49.691968 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc417726-1f8b-4510-82e7-882bf6c02d37" containerName="payload-pre-processing" Sep 18 07:09:49.694412 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:49.694397 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-87c5n/perf-node-gather-daemonset-ntpcl" Sep 18 07:09:49.700711 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:49.700445 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-87c5n\"/\"kube-root-ca.crt\"" Sep 18 07:09:49.701678 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:49.701658 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-87c5n\"/\"default-dockercfg-h95g6\"" Sep 18 07:09:49.701804 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:49.701681 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-87c5n\"/\"openshift-service-ca.crt\"" Sep 18 07:09:49.728668 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:49.728639 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-87c5n/perf-node-gather-daemonset-ntpcl"] Sep 18 07:09:49.847982 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:49.847946 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/173740ba-0c6e-4c25-b5d4-fbec67cb7ad1-sys\") pod \"perf-node-gather-daemonset-ntpcl\" (UID: \"173740ba-0c6e-4c25-b5d4-fbec67cb7ad1\") " pod="openshift-must-gather-87c5n/perf-node-gather-daemonset-ntpcl" Sep 18 07:09:49.848185 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:49.848001 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh7zb\" (UniqueName: \"kubernetes.io/projected/173740ba-0c6e-4c25-b5d4-fbec67cb7ad1-kube-api-access-xh7zb\") pod \"perf-node-gather-daemonset-ntpcl\" (UID: \"173740ba-0c6e-4c25-b5d4-fbec67cb7ad1\") " pod="openshift-must-gather-87c5n/perf-node-gather-daemonset-ntpcl" Sep 18 07:09:49.848185 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:49.848057 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/173740ba-0c6e-4c25-b5d4-fbec67cb7ad1-podres\") pod \"perf-node-gather-daemonset-ntpcl\" (UID: \"173740ba-0c6e-4c25-b5d4-fbec67cb7ad1\") " pod="openshift-must-gather-87c5n/perf-node-gather-daemonset-ntpcl" Sep 18 07:09:49.848185 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:49.848089 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/173740ba-0c6e-4c25-b5d4-fbec67cb7ad1-lib-modules\") pod \"perf-node-gather-daemonset-ntpcl\" (UID: \"173740ba-0c6e-4c25-b5d4-fbec67cb7ad1\") " pod="openshift-must-gather-87c5n/perf-node-gather-daemonset-ntpcl" Sep 18 07:09:49.848185 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:49.848128 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/173740ba-0c6e-4c25-b5d4-fbec67cb7ad1-proc\") pod \"perf-node-gather-daemonset-ntpcl\" (UID: \"173740ba-0c6e-4c25-b5d4-fbec67cb7ad1\") " pod="openshift-must-gather-87c5n/perf-node-gather-daemonset-ntpcl" Sep 18 07:09:49.949416 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:49.949294 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/173740ba-0c6e-4c25-b5d4-fbec67cb7ad1-sys\") pod \"perf-node-gather-daemonset-ntpcl\" (UID: \"173740ba-0c6e-4c25-b5d4-fbec67cb7ad1\") " pod="openshift-must-gather-87c5n/perf-node-gather-daemonset-ntpcl" Sep 18 07:09:49.949416 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:49.949382 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh7zb\" (UniqueName: \"kubernetes.io/projected/173740ba-0c6e-4c25-b5d4-fbec67cb7ad1-kube-api-access-xh7zb\") pod \"perf-node-gather-daemonset-ntpcl\" (UID: \"173740ba-0c6e-4c25-b5d4-fbec67cb7ad1\") " pod="openshift-must-gather-87c5n/perf-node-gather-daemonset-ntpcl" Sep 18 07:09:49.949674 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:49.949446 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/173740ba-0c6e-4c25-b5d4-fbec67cb7ad1-sys\") pod \"perf-node-gather-daemonset-ntpcl\" (UID: \"173740ba-0c6e-4c25-b5d4-fbec67cb7ad1\") " pod="openshift-must-gather-87c5n/perf-node-gather-daemonset-ntpcl" Sep 18 07:09:49.949674 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:49.949463 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/173740ba-0c6e-4c25-b5d4-fbec67cb7ad1-podres\") pod \"perf-node-gather-daemonset-ntpcl\" (UID: \"173740ba-0c6e-4c25-b5d4-fbec67cb7ad1\") " pod="openshift-must-gather-87c5n/perf-node-gather-daemonset-ntpcl" Sep 18 07:09:49.949674 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:49.949506 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/173740ba-0c6e-4c25-b5d4-fbec67cb7ad1-lib-modules\") pod \"perf-node-gather-daemonset-ntpcl\" (UID: \"173740ba-0c6e-4c25-b5d4-fbec67cb7ad1\") " pod="openshift-must-gather-87c5n/perf-node-gather-daemonset-ntpcl" Sep 18 07:09:49.949674 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:49.949553 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/173740ba-0c6e-4c25-b5d4-fbec67cb7ad1-proc\") pod \"perf-node-gather-daemonset-ntpcl\" (UID: \"173740ba-0c6e-4c25-b5d4-fbec67cb7ad1\") " pod="openshift-must-gather-87c5n/perf-node-gather-daemonset-ntpcl" Sep 18 07:09:49.949674 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:49.949634 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/173740ba-0c6e-4c25-b5d4-fbec67cb7ad1-podres\") pod \"perf-node-gather-daemonset-ntpcl\" (UID: \"173740ba-0c6e-4c25-b5d4-fbec67cb7ad1\") " pod="openshift-must-gather-87c5n/perf-node-gather-daemonset-ntpcl" Sep 18 07:09:49.949853 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:49.949675 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/173740ba-0c6e-4c25-b5d4-fbec67cb7ad1-lib-modules\") pod \"perf-node-gather-daemonset-ntpcl\" (UID: \"173740ba-0c6e-4c25-b5d4-fbec67cb7ad1\") " pod="openshift-must-gather-87c5n/perf-node-gather-daemonset-ntpcl" Sep 18 07:09:49.949853 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:49.949715 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/173740ba-0c6e-4c25-b5d4-fbec67cb7ad1-proc\") pod \"perf-node-gather-daemonset-ntpcl\" (UID: \"173740ba-0c6e-4c25-b5d4-fbec67cb7ad1\") " pod="openshift-must-gather-87c5n/perf-node-gather-daemonset-ntpcl" Sep 18 07:09:49.962096 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:49.962068 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh7zb\" (UniqueName: \"kubernetes.io/projected/173740ba-0c6e-4c25-b5d4-fbec67cb7ad1-kube-api-access-xh7zb\") pod \"perf-node-gather-daemonset-ntpcl\" (UID: \"173740ba-0c6e-4c25-b5d4-fbec67cb7ad1\") " pod="openshift-must-gather-87c5n/perf-node-gather-daemonset-ntpcl" Sep 18 07:09:50.004206 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:50.004164 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-87c5n/perf-node-gather-daemonset-ntpcl" Sep 18 07:09:50.283472 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:50.283441 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-87c5n/perf-node-gather-daemonset-ntpcl"] Sep 18 07:09:50.284520 ip-10-0-142-104 kubenswrapper[2567]: W0918 07:09:50.284491 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod173740ba_0c6e_4c25_b5d4_fbec67cb7ad1.slice/crio-5994b277002d77c3bb869d34386923ae5d5aa8bc0202b802ff2c1c3ecdcb04f0 WatchSource:0}: Error finding container 5994b277002d77c3bb869d34386923ae5d5aa8bc0202b802ff2c1c3ecdcb04f0: Status 404 returned error can't find the container with id 5994b277002d77c3bb869d34386923ae5d5aa8bc0202b802ff2c1c3ecdcb04f0 Sep 18 07:09:50.807804 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:50.807763 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-87c5n/perf-node-gather-daemonset-ntpcl" event={"ID":"173740ba-0c6e-4c25-b5d4-fbec67cb7ad1","Type":"ContainerStarted","Data":"13dfc0e06a802b723361191918a930f514a85d845fcc04012f7649e3b249fd90"} Sep 18 07:09:50.808207 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:50.807827 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-87c5n/perf-node-gather-daemonset-ntpcl" event={"ID":"173740ba-0c6e-4c25-b5d4-fbec67cb7ad1","Type":"ContainerStarted","Data":"5994b277002d77c3bb869d34386923ae5d5aa8bc0202b802ff2c1c3ecdcb04f0"} Sep 18 07:09:50.808207 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:50.807856 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-87c5n/perf-node-gather-daemonset-ntpcl" Sep 18 07:09:50.832783 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:50.832732 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-87c5n/perf-node-gather-daemonset-ntpcl" podStartSLOduration=1.832715421 podStartE2EDuration="1.832715421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 07:09:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 07:09:50.831224561 +0000 UTC m=+2852.570640758" watchObservedRunningTime="2026-09-18 07:09:50.832715421 +0000 UTC m=+2852.572131595" Sep 18 07:09:51.288689 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:51.288660 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-fthpb_6164b435-ee87-4137-b551-f951321399e6/volume-data-source-validator/0.log" Sep 18 07:09:52.775927 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:52.775897 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ph66s_8c9f31bd-99bd-4a50-8d28-72dfc56290ae/dns/0.log" Sep 18 07:09:52.812208 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:52.812181 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ph66s_8c9f31bd-99bd-4a50-8d28-72dfc56290ae/kube-rbac-proxy/0.log" Sep 18 07:09:52.865106 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:52.865075 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kgx9w_c7e27a94-0a48-466c-abdb-9f6b0758e358/dns-node-resolver/0.log" Sep 18 07:09:53.773869 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:53.773838 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xccgg_27cccfd2-ff8e-4f71-bc24-5072f09b7f21/node-ca/0.log" Sep 18 07:09:54.854632 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:54.854596 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg_2a4b02c0-6cda-40d5-a37a-05f0ba6ea4b4/istio-proxy/0.log" Sep 18 07:09:55.299399 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:55.299277 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-rzl5d_87337e45-0211-4fb0-bf69-7b19c2934525/discovery/0.log" Sep 18 07:09:55.496535 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:55.496508 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-jbzm5_96a61b61-7de1-428e-9fb6-99d5534fb7db/payload-processing/0.log" Sep 18 07:09:55.528246 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:55.528219 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-75b6d7b44b-fvj8r_e16d183f-ea45-4a4c-94be-f078321c3c79/router/0.log" Sep 18 07:09:56.477777 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:56.477748 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qxhxp_3320df23-25b7-4e16-9024-076401f7ca3e/serve-healthcheck-canary/0.log" Sep 18 07:09:56.823128 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:56.823054 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-87c5n/perf-node-gather-daemonset-ntpcl" Sep 18 07:09:57.111762 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:57.111730 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-5bn97_ee4fa404-921d-48d8-a26d-fa5c15425928/insights-operator/0.log" Sep 18 07:09:57.112103 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:57.112091 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-5bn97_ee4fa404-921d-48d8-a26d-fa5c15425928/insights-operator/1.log" Sep 18 07:09:57.229724 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:57.229685 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6nf5m_b5029a64-fc57-44f7-b2e8-79c9a82c572d/kube-rbac-proxy/0.log" Sep 18 07:09:57.255830 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:57.255799 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6nf5m_b5029a64-fc57-44f7-b2e8-79c9a82c572d/exporter/0.log" Sep 18 07:09:57.283465 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:57.283440 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6nf5m_b5029a64-fc57-44f7-b2e8-79c9a82c572d/extractor/0.log" Sep 18 07:09:59.803126 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:09:59.803091 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg_2a4b02c0-6cda-40d5-a37a-05f0ba6ea4b4/istio-proxy/0.log" Sep 18 07:10:00.263409 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:10:00.263355 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-rzl5d_87337e45-0211-4fb0-bf69-7b19c2934525/discovery/0.log" Sep 18 07:10:00.481398 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:10:00.481293 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-jbzm5_96a61b61-7de1-428e-9fb6-99d5534fb7db/payload-processing/0.log" Sep 18 07:10:00.507070 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:10:00.507032 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-75b6d7b44b-fvj8r_e16d183f-ea45-4a4c-94be-f078321c3c79/router/0.log" Sep 18 07:10:00.554095 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:10:00.554062 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-qgwmc_61e31de6-574c-43c5-a839-9dd1d9067587/manager/0.log" Sep 18 07:10:00.741897 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:10:00.741808 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6fd575c64f-7mcbn_44103c01-51b0-4d76-95e5-cb3b273f78ab/manager/0.log" Sep 18 07:10:00.806467 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:10:00.806443 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6fd575c64f-7mcbn_44103c01-51b0-4d76-95e5-cb3b273f78ab/copy-manifests/0.log" Sep 18 07:10:02.236870 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:10:02.236834 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-6bslg_2a4b02c0-6cda-40d5-a37a-05f0ba6ea4b4/istio-proxy/0.log" Sep 18 07:10:02.455708 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:10:02.455678 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-64c86f57d4-cmwvd_74420ed8-08f5-471f-9681-5558063a090c/manager/0.log" Sep 18 07:10:02.684751 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:10:02.684717 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-rzl5d_87337e45-0211-4fb0-bf69-7b19c2934525/discovery/0.log" Sep 18 07:10:02.829263 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:10:02.829176 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-jbzm5_96a61b61-7de1-428e-9fb6-99d5534fb7db/payload-processing/0.log" Sep 18 07:10:02.854108 ip-10-0-142-104 kubenswrapper[2567]: I0918 07:10:02.854069 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-75b6d7b44b-fvj8r_e16d183f-ea45-4a4c-94be-f078321c3c79/router/0.log"