Sep 07 16:19:29.761204 ip-10-0-135-242 systemd[1]: Starting Kubernetes Kubelet... Sep 07 16:19:30.157096 ip-10-0-135-242 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 07 16:19:30.157096 ip-10-0-135-242 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 07 16:19:30.157096 ip-10-0-135-242 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 07 16:19:30.157096 ip-10-0-135-242 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 07 16:19:30.157096 ip-10-0-135-242 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 07 16:19:30.157986 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.157878 2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 07 16:19:30.161498 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161477 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 07 16:19:30.161498 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161496 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 07 16:19:30.161498 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161502 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 07 16:19:30.161498 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161507 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 07 16:19:30.161753 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161511 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 07 16:19:30.161753 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161516 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 07 16:19:30.161753 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161520 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 07 16:19:30.161753 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161523 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 07 16:19:30.161753 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161528 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 07 16:19:30.161753 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161532 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 07 16:19:30.161753 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161545 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 07 16:19:30.161753 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161549 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 07 16:19:30.161753 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161553 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 07 16:19:30.161753 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161559 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 07 16:19:30.161753 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161563 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 07 16:19:30.161753 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161567 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 07 16:19:30.161753 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161571 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 07 16:19:30.161753 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161575 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 07 16:19:30.161753 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161579 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 07 16:19:30.161753 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161584 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 07 16:19:30.161753 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161587 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 07 16:19:30.161753 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161591 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 07 16:19:30.161753 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161595 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 07 16:19:30.162593 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161599 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 07 16:19:30.162593 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161603 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 07 16:19:30.162593 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161608 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 07 16:19:30.162593 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161613 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 07 16:19:30.162593 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161617 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 07 16:19:30.162593 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161621 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 07 16:19:30.162593 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161625 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 07 16:19:30.162593 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161629 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 07 16:19:30.162593 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161634 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 07 16:19:30.162593 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161639 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 07 16:19:30.162593 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161643 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 07 16:19:30.162593 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161648 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 07 16:19:30.162593 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161652 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 07 16:19:30.162593 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161658 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 07 16:19:30.162593 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161663 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 07 16:19:30.162593 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161667 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 07 16:19:30.162593 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161671 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 07 16:19:30.162593 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161675 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 07 16:19:30.162593 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161679 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 07 16:19:30.162593 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161684 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 07 16:19:30.163463 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161690 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 07 16:19:30.163463 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161697 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 07 16:19:30.163463 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161701 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 07 16:19:30.163463 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161706 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 07 16:19:30.163463 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161710 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 07 16:19:30.163463 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161714 2580 feature_gate.go:328] unrecognized feature gate: Example2 Sep 07 16:19:30.163463 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161718 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 07 16:19:30.163463 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161723 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 07 16:19:30.163463 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161727 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 07 16:19:30.163463 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161731 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 07 16:19:30.163463 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161736 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 07 16:19:30.163463 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161740 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 07 16:19:30.163463 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161744 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 07 16:19:30.163463 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161748 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 07 16:19:30.163463 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161752 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 07 16:19:30.163463 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161756 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 07 16:19:30.163463 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161760 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 07 16:19:30.163463 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161764 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 07 16:19:30.163463 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161768 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 07 16:19:30.164245 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161772 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 07 16:19:30.164245 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161776 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 07 16:19:30.164245 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161781 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 07 16:19:30.164245 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161786 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 07 16:19:30.164245 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161791 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 07 16:19:30.164245 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161795 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 07 16:19:30.164245 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161799 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 07 16:19:30.164245 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161804 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 07 16:19:30.164245 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161808 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 07 16:19:30.164245 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161812 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 07 16:19:30.164245 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161818 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 07 16:19:30.164245 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161822 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 07 16:19:30.164245 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161826 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 07 16:19:30.164245 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161830 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 07 16:19:30.164245 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161834 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 07 16:19:30.164245 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161838 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 07 16:19:30.164245 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161843 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 07 16:19:30.164245 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161846 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 07 16:19:30.164245 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161850 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 07 16:19:30.164245 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161857 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 07 16:19:30.165103 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161861 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 07 16:19:30.165103 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161866 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 07 16:19:30.165103 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161870 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 07 16:19:30.165103 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161874 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 07 16:19:30.165103 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161879 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 07 16:19:30.165103 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161883 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 07 16:19:30.165103 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161887 2580 feature_gate.go:328] unrecognized feature gate: Example Sep 07 16:19:30.165103 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161907 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 07 16:19:30.165103 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161911 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 07 16:19:30.165103 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161915 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 07 16:19:30.165103 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161920 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 07 16:19:30.165103 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161923 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 07 16:19:30.165103 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161928 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 07 16:19:30.165103 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161932 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 07 16:19:30.165103 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161936 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 07 16:19:30.165103 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161942 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 07 16:19:30.165103 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.161947 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 07 16:19:30.165103 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163117 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 07 16:19:30.165103 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163128 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 07 16:19:30.165103 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163133 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 07 16:19:30.165986 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163139 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 07 16:19:30.165986 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163143 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 07 16:19:30.165986 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163147 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 07 16:19:30.165986 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163151 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 07 16:19:30.165986 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163156 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 07 16:19:30.165986 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163160 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 07 16:19:30.165986 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163164 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 07 16:19:30.165986 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163168 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 07 16:19:30.165986 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163172 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 07 16:19:30.165986 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163176 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 07 16:19:30.165986 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163181 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 07 16:19:30.165986 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163185 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 07 16:19:30.165986 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163189 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 07 16:19:30.165986 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163193 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 07 16:19:30.165986 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163197 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 07 16:19:30.165986 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163201 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 07 16:19:30.165986 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163206 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 07 16:19:30.165986 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163209 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 07 16:19:30.165986 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163217 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 07 16:19:30.165986 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163223 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 07 16:19:30.166675 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163228 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 07 16:19:30.166675 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163233 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 07 16:19:30.166675 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163237 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 07 16:19:30.166675 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163242 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 07 16:19:30.166675 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163247 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 07 16:19:30.166675 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163251 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 07 16:19:30.166675 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163255 2580 feature_gate.go:328] unrecognized feature gate: Example Sep 07 16:19:30.166675 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163260 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 07 16:19:30.166675 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163266 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 07 16:19:30.166675 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163271 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 07 16:19:30.166675 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163275 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 07 16:19:30.166675 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163280 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 07 16:19:30.166675 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163284 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 07 16:19:30.166675 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163288 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 07 16:19:30.166675 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163292 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 07 16:19:30.166675 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163296 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 07 16:19:30.166675 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163301 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 07 16:19:30.166675 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163305 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 07 16:19:30.166675 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163311 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 07 16:19:30.166675 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163317 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 07 16:19:30.167280 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163322 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 07 16:19:30.167280 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163326 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 07 16:19:30.167280 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163330 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 07 16:19:30.167280 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163335 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 07 16:19:30.167280 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163339 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 07 16:19:30.167280 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163345 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 07 16:19:30.167280 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163349 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 07 16:19:30.167280 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163354 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 07 16:19:30.167280 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163358 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 07 16:19:30.167280 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163362 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 07 16:19:30.167280 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163366 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 07 16:19:30.167280 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163373 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 07 16:19:30.167280 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163376 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 07 16:19:30.167280 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163381 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 07 16:19:30.167280 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163385 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 07 16:19:30.167280 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163390 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 07 16:19:30.167280 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163394 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 07 16:19:30.167280 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163399 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 07 16:19:30.167280 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163403 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 07 16:19:30.167280 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163408 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 07 16:19:30.167800 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163412 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 07 16:19:30.167800 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163417 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 07 16:19:30.167800 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163421 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 07 16:19:30.167800 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163425 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 07 16:19:30.167800 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163430 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 07 16:19:30.167800 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163433 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 07 16:19:30.167800 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163437 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 07 16:19:30.167800 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163442 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 07 16:19:30.167800 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163446 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 07 16:19:30.167800 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163450 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 07 16:19:30.167800 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163454 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 07 16:19:30.167800 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163458 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 07 16:19:30.167800 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163462 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 07 16:19:30.167800 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163466 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 07 16:19:30.167800 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163470 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 07 16:19:30.167800 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163478 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 07 16:19:30.167800 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163484 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 07 16:19:30.167800 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163488 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 07 16:19:30.167800 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163493 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 07 16:19:30.167800 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163497 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 07 16:19:30.168406 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163502 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 07 16:19:30.168406 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163506 2580 feature_gate.go:328] unrecognized feature gate: Example2 Sep 07 16:19:30.168406 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163510 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 07 16:19:30.168406 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163514 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 07 16:19:30.168406 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163518 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 07 16:19:30.168406 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163522 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 07 16:19:30.168406 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163526 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 07 16:19:30.168406 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163530 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 07 16:19:30.168406 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163534 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 07 16:19:30.168406 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163538 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 07 16:19:30.168406 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163542 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 07 16:19:30.168406 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163546 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 07 16:19:30.168406 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163551 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 07 16:19:30.168406 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163555 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 07 16:19:30.168406 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163561 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 07 16:19:30.168406 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.163565 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 07 16:19:30.168406 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164226 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 07 16:19:30.168406 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164234 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 07 16:19:30.168406 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164239 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 07 16:19:30.169021 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164243 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 07 16:19:30.169021 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164246 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 07 16:19:30.169021 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164250 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 07 16:19:30.169021 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164254 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 07 16:19:30.169021 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164259 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 07 16:19:30.169021 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164263 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 07 16:19:30.169021 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164268 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 07 16:19:30.169021 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164272 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 07 16:19:30.169021 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164276 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 07 16:19:30.169021 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164280 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 07 16:19:30.169021 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164286 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 07 16:19:30.169021 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164293 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 07 16:19:30.169021 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164298 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 07 16:19:30.169021 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164302 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 07 16:19:30.169021 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164307 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 07 16:19:30.169021 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164311 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 07 16:19:30.169021 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164315 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 07 16:19:30.169021 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164320 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 07 16:19:30.169021 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164324 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 07 16:19:30.169021 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164328 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 07 16:19:30.169527 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164332 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 07 16:19:30.169527 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164337 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 07 16:19:30.169527 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164341 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 07 16:19:30.169527 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164345 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 07 16:19:30.169527 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164349 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 07 16:19:30.169527 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164354 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 07 16:19:30.169527 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164357 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 07 16:19:30.169527 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164363 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 07 16:19:30.169527 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164368 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 07 16:19:30.169527 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164372 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 07 16:19:30.169527 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164377 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 07 16:19:30.169527 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164381 2580 feature_gate.go:328] unrecognized feature gate: Example Sep 07 16:19:30.169527 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164385 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 07 16:19:30.169527 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164390 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 07 16:19:30.169527 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164393 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 07 16:19:30.169527 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164398 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 07 16:19:30.169527 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164403 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 07 16:19:30.169527 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164407 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 07 16:19:30.169527 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164411 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 07 16:19:30.169527 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164415 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 07 16:19:30.170126 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164419 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 07 16:19:30.170126 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164423 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 07 16:19:30.170126 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164427 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 07 16:19:30.170126 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164431 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 07 16:19:30.170126 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164435 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 07 16:19:30.170126 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164439 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 07 16:19:30.170126 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164444 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 07 16:19:30.170126 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164448 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 07 16:19:30.170126 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164452 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 07 16:19:30.170126 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164456 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 07 16:19:30.170126 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164460 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 07 16:19:30.170126 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164464 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 07 16:19:30.170126 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164469 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 07 16:19:30.170126 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164474 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 07 16:19:30.170126 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164478 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 07 16:19:30.170126 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164484 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 07 16:19:30.170126 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164491 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 07 16:19:30.170126 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164497 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 07 16:19:30.170126 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164503 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 07 16:19:30.170126 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164507 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 07 16:19:30.170638 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164512 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 07 16:19:30.170638 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164516 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 07 16:19:30.170638 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164520 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 07 16:19:30.170638 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164526 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 07 16:19:30.170638 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164530 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 07 16:19:30.170638 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164534 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 07 16:19:30.170638 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164538 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 07 16:19:30.170638 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164543 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 07 16:19:30.170638 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164547 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 07 16:19:30.170638 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164551 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 07 16:19:30.170638 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164555 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 07 16:19:30.170638 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164559 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 07 16:19:30.170638 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164563 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 07 16:19:30.170638 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164567 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 07 16:19:30.170638 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164571 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 07 16:19:30.170638 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164575 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 07 16:19:30.170638 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164579 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 07 16:19:30.170638 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164583 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 07 16:19:30.170638 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164588 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 07 16:19:30.170638 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164592 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 07 16:19:30.171160 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164598 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 07 16:19:30.171160 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164604 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 07 16:19:30.171160 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164608 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 07 16:19:30.171160 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164613 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 07 16:19:30.171160 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164618 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 07 16:19:30.171160 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164623 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 07 16:19:30.171160 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164627 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 07 16:19:30.171160 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164630 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 07 16:19:30.171160 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164637 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 07 16:19:30.171160 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164642 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 07 16:19:30.171160 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164645 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 07 16:19:30.171160 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164649 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 07 16:19:30.171160 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164654 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 07 16:19:30.171160 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164659 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 07 16:19:30.171160 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164663 2580 feature_gate.go:328] unrecognized feature gate: Example2 Sep 07 16:19:30.171160 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.164667 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 07 16:19:30.171160 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.164779 2580 flags.go:64] FLAG: --address="0.0.0.0" Sep 07 16:19:30.171160 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.164829 2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 07 16:19:30.171160 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.164839 2580 flags.go:64] FLAG: --anonymous-auth="true" Sep 07 16:19:30.171160 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.164846 2580 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 07 16:19:30.171651 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.164853 2580 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 07 16:19:30.171651 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.164858 2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 07 16:19:30.171651 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.164865 2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 07 16:19:30.171651 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.164872 2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 07 16:19:30.171651 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.164876 2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 07 16:19:30.171651 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.164881 2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 07 16:19:30.171651 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.164887 2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 07 16:19:30.171651 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.164911 2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 07 16:19:30.171651 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.164916 2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 07 16:19:30.171651 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.164924 2580 flags.go:64] FLAG: --cgroup-root="" Sep 07 16:19:30.171651 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.164929 2580 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 07 16:19:30.171651 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.164934 2580 flags.go:64] FLAG: --client-ca-file="" Sep 07 16:19:30.171651 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.164939 2580 flags.go:64] FLAG: --cloud-provider="external" Sep 07 16:19:30.171651 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.164943 2580 flags.go:64] FLAG: --cluster-dns="[]" Sep 07 16:19:30.171651 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.164949 2580 flags.go:64] FLAG: --cluster-domain="" Sep 07 16:19:30.171651 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.164954 2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 07 16:19:30.171651 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.164959 2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 07 16:19:30.171651 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.164964 2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 07 16:19:30.171651 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.164970 2580 flags.go:64] FLAG: --container-log-max-files="5" Sep 07 16:19:30.171651 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.164976 2580 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 07 16:19:30.171651 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.164982 2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 07 16:19:30.171651 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.164987 2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 07 16:19:30.171651 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.164993 2580 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 07 16:19:30.172219 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.164998 2580 flags.go:64] FLAG: --contention-profiling="false" Sep 07 16:19:30.172219 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165002 2580 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 07 16:19:30.172219 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165007 2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 07 16:19:30.172219 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165013 2580 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 07 16:19:30.172219 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165018 2580 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 07 16:19:30.172219 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165024 2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 07 16:19:30.172219 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165029 2580 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 07 16:19:30.172219 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165034 2580 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 07 16:19:30.172219 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165038 2580 flags.go:64] FLAG: --enable-load-reader="false" Sep 07 16:19:30.172219 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165043 2580 flags.go:64] FLAG: --enable-server="true" Sep 07 16:19:30.172219 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165048 2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 07 16:19:30.172219 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165055 2580 flags.go:64] FLAG: --event-burst="100" Sep 07 16:19:30.172219 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165060 2580 flags.go:64] FLAG: --event-qps="50" Sep 07 16:19:30.172219 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165064 2580 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 07 16:19:30.172219 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165070 2580 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 07 16:19:30.172219 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165075 2580 flags.go:64] FLAG: --eviction-hard="" Sep 07 16:19:30.172219 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165081 2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 07 16:19:30.172219 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165086 2580 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 07 16:19:30.172219 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165093 2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 07 16:19:30.172219 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165097 2580 flags.go:64] FLAG: --eviction-soft="" Sep 07 16:19:30.172219 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165102 2580 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 07 16:19:30.172219 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165107 2580 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 07 16:19:30.172219 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165111 2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 07 16:19:30.172219 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165116 2580 flags.go:64] FLAG: --experimental-mounter-path="" Sep 07 16:19:30.172219 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165120 2580 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 07 16:19:30.172966 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165124 2580 flags.go:64] FLAG: --fail-swap-on="true" Sep 07 16:19:30.172966 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165129 2580 flags.go:64] FLAG: --feature-gates="" Sep 07 16:19:30.172966 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165135 2580 flags.go:64] FLAG: --file-check-frequency="20s" Sep 07 16:19:30.172966 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165147 2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 07 16:19:30.172966 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165155 2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 07 16:19:30.172966 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165160 2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 07 16:19:30.172966 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165165 2580 flags.go:64] FLAG: --healthz-port="10248" Sep 07 16:19:30.172966 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165170 2580 flags.go:64] FLAG: --help="false" Sep 07 16:19:30.172966 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165175 2580 flags.go:64] FLAG: --hostname-override="ip-10-0-135-242.ec2.internal" Sep 07 16:19:30.172966 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165180 2580 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 07 16:19:30.172966 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165184 2580 flags.go:64] FLAG: --http-check-frequency="20s" Sep 07 16:19:30.172966 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165190 2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 07 16:19:30.172966 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165196 2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 07 16:19:30.172966 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165201 2580 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 07 16:19:30.172966 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165206 2580 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 07 16:19:30.172966 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165210 2580 flags.go:64] FLAG: --image-service-endpoint="" Sep 07 16:19:30.172966 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165215 2580 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 07 16:19:30.172966 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165220 2580 flags.go:64] FLAG: --kube-api-burst="100" Sep 07 16:19:30.172966 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165224 2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 07 16:19:30.172966 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165230 2580 flags.go:64] FLAG: --kube-api-qps="50" Sep 07 16:19:30.172966 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165234 2580 flags.go:64] FLAG: --kube-reserved="" Sep 07 16:19:30.172966 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165239 2580 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 07 16:19:30.172966 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165244 2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 07 16:19:30.172966 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165249 2580 flags.go:64] FLAG: --kubelet-cgroups="" Sep 07 16:19:30.173561 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165253 2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 07 16:19:30.173561 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165260 2580 flags.go:64] FLAG: --lock-file="" Sep 07 16:19:30.173561 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165264 2580 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 07 16:19:30.173561 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165269 2580 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 07 16:19:30.173561 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165273 2580 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 07 16:19:30.173561 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165283 2580 flags.go:64] FLAG: --log-json-split-stream="false" Sep 07 16:19:30.173561 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165288 2580 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 07 16:19:30.173561 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165292 2580 flags.go:64] FLAG: --log-text-split-stream="false" Sep 07 16:19:30.173561 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165297 2580 flags.go:64] FLAG: --logging-format="text" Sep 07 16:19:30.173561 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165301 2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 07 16:19:30.173561 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165306 2580 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 07 16:19:30.173561 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165311 2580 flags.go:64] FLAG: --manifest-url="" Sep 07 16:19:30.173561 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165317 2580 flags.go:64] FLAG: --manifest-url-header="" Sep 07 16:19:30.173561 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165324 2580 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 07 16:19:30.173561 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165329 2580 flags.go:64] FLAG: --max-open-files="1000000" Sep 07 16:19:30.173561 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165335 2580 flags.go:64] FLAG: --max-pods="110" Sep 07 16:19:30.173561 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165340 2580 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 07 16:19:30.173561 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165345 2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 07 16:19:30.173561 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165349 2580 flags.go:64] FLAG: --memory-manager-policy="None" Sep 07 16:19:30.173561 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165354 2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 07 16:19:30.173561 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165359 2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 07 16:19:30.173561 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165365 2580 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 07 16:19:30.173561 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165370 2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 07 16:19:30.173561 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165380 2580 flags.go:64] FLAG: --node-status-max-images="50" Sep 07 16:19:30.173561 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165385 2580 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 07 16:19:30.174399 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165390 2580 flags.go:64] FLAG: --oom-score-adj="-999" Sep 07 16:19:30.174399 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165395 2580 flags.go:64] FLAG: --pod-cidr="" Sep 07 16:19:30.174399 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165400 2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 07 16:19:30.174399 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165409 2580 flags.go:64] FLAG: --pod-manifest-path="" Sep 07 16:19:30.174399 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165413 2580 flags.go:64] FLAG: --pod-max-pids="-1" Sep 07 16:19:30.174399 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165418 2580 flags.go:64] FLAG: --pods-per-core="0" Sep 07 16:19:30.174399 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165422 2580 flags.go:64] FLAG: --port="10250" Sep 07 16:19:30.174399 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165427 2580 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 07 16:19:30.174399 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165434 2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0114240b7b01fcf07" Sep 07 16:19:30.174399 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165439 2580 flags.go:64] FLAG: --qos-reserved="" Sep 07 16:19:30.174399 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165443 2580 flags.go:64] FLAG: --read-only-port="10255" Sep 07 16:19:30.174399 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165448 2580 flags.go:64] FLAG: --register-node="true" Sep 07 16:19:30.174399 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165453 2580 flags.go:64] FLAG: --register-with-taints="" Sep 07 16:19:30.174399 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165459 2580 flags.go:64] FLAG: --registry-burst="10" Sep 07 16:19:30.174399 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165463 2580 flags.go:64] FLAG: --registry-qps="5" Sep 07 16:19:30.174399 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165468 2580 flags.go:64] FLAG: --reserved-cpus="" Sep 07 16:19:30.174399 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165472 2580 flags.go:64] FLAG: --reserved-memory="" Sep 07 16:19:30.174399 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165477 2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 07 16:19:30.174399 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165483 2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 07 16:19:30.174399 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165488 2580 flags.go:64] FLAG: --rotate-certificates="false" Sep 07 16:19:30.174399 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165494 2580 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 07 16:19:30.174399 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165499 2580 flags.go:64] FLAG: --runonce="false" Sep 07 16:19:30.174399 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165504 2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 07 16:19:30.174399 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165509 2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 07 16:19:30.174399 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165513 2580 flags.go:64] FLAG: --seccomp-default="false" Sep 07 16:19:30.175286 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165517 2580 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 07 16:19:30.175286 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165522 2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 07 16:19:30.175286 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165527 2580 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 07 16:19:30.175286 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165533 2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 07 16:19:30.175286 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165538 2580 flags.go:64] FLAG: --storage-driver-password="root" Sep 07 16:19:30.175286 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165543 2580 flags.go:64] FLAG: --storage-driver-secure="false" Sep 07 16:19:30.175286 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165547 2580 flags.go:64] FLAG: --storage-driver-table="stats" Sep 07 16:19:30.175286 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165552 2580 flags.go:64] FLAG: --storage-driver-user="root" Sep 07 16:19:30.175286 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165556 2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 07 16:19:30.175286 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165561 2580 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 07 16:19:30.175286 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165566 2580 flags.go:64] FLAG: --system-cgroups="" Sep 07 16:19:30.175286 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165571 2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 07 16:19:30.175286 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165579 2580 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 07 16:19:30.175286 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165583 2580 flags.go:64] FLAG: --tls-cert-file="" Sep 07 16:19:30.175286 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165588 2580 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 07 16:19:30.175286 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165595 2580 flags.go:64] FLAG: --tls-min-version="" Sep 07 16:19:30.175286 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165600 2580 flags.go:64] FLAG: --tls-private-key-file="" Sep 07 16:19:30.175286 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165604 2580 flags.go:64] FLAG: --topology-manager-policy="none" Sep 07 16:19:30.175286 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165609 2580 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 07 16:19:30.175286 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165614 2580 flags.go:64] FLAG: --topology-manager-scope="container" Sep 07 16:19:30.175286 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165619 2580 flags.go:64] FLAG: --v="2" Sep 07 16:19:30.175286 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165625 2580 flags.go:64] FLAG: --version="false" Sep 07 16:19:30.175286 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165631 2580 flags.go:64] FLAG: --vmodule="" Sep 07 16:19:30.175286 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165644 2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 07 16:19:30.175286 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.165650 2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 07 16:19:30.175915 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.165846 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 07 16:19:30.175915 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.165853 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 07 16:19:30.175915 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.165859 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 07 16:19:30.175915 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.165863 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 07 16:19:30.175915 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.165868 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 07 16:19:30.175915 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.165873 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 07 16:19:30.175915 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.165878 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 07 16:19:30.175915 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.165883 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 07 16:19:30.175915 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.165887 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 07 16:19:30.175915 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.165909 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 07 16:19:30.175915 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.165913 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 07 16:19:30.175915 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.165919 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 07 16:19:30.175915 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.165923 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 07 16:19:30.175915 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.165927 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 07 16:19:30.175915 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.165931 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 07 16:19:30.175915 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.165935 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 07 16:19:30.175915 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.165939 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 07 16:19:30.175915 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.165943 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 07 16:19:30.175915 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.165947 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 07 16:19:30.175915 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.165951 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 07 16:19:30.176448 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.165955 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 07 16:19:30.176448 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.165960 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 07 16:19:30.176448 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.165967 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 07 16:19:30.176448 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.165974 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 07 16:19:30.176448 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.165979 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 07 16:19:30.176448 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.165984 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 07 16:19:30.176448 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.165989 2580 feature_gate.go:328] unrecognized feature gate: Example2 Sep 07 16:19:30.176448 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.165993 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 07 16:19:30.176448 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.165997 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 07 16:19:30.176448 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166001 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 07 16:19:30.176448 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166005 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 07 16:19:30.176448 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166009 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 07 16:19:30.176448 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166013 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 07 16:19:30.176448 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166017 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 07 16:19:30.176448 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166024 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 07 16:19:30.176448 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166029 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 07 16:19:30.176448 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166033 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 07 16:19:30.176448 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166037 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 07 16:19:30.176448 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166041 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 07 16:19:30.176934 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166045 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 07 16:19:30.176934 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166049 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 07 16:19:30.176934 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166054 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 07 16:19:30.176934 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166058 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 07 16:19:30.176934 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166062 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 07 16:19:30.176934 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166067 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 07 16:19:30.176934 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166071 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 07 16:19:30.176934 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166076 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 07 16:19:30.176934 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166080 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 07 16:19:30.176934 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166084 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 07 16:19:30.176934 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166088 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 07 16:19:30.176934 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166092 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 07 16:19:30.176934 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166096 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 07 16:19:30.176934 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166100 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 07 16:19:30.176934 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166104 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 07 16:19:30.176934 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166110 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 07 16:19:30.176934 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166114 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 07 16:19:30.176934 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166119 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 07 16:19:30.176934 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166123 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 07 16:19:30.176934 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166127 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 07 16:19:30.177445 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166131 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 07 16:19:30.177445 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166135 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 07 16:19:30.177445 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166139 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 07 16:19:30.177445 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166143 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 07 16:19:30.177445 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166147 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 07 16:19:30.177445 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166151 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 07 16:19:30.177445 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166155 2580 feature_gate.go:328] unrecognized feature gate: Example Sep 07 16:19:30.177445 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166161 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 07 16:19:30.177445 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166165 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 07 16:19:30.177445 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166169 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 07 16:19:30.177445 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166173 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 07 16:19:30.177445 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166177 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 07 16:19:30.177445 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166181 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 07 16:19:30.177445 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166185 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 07 16:19:30.177445 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166190 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 07 16:19:30.177445 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166194 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 07 16:19:30.177445 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166198 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 07 16:19:30.177445 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166203 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 07 16:19:30.177445 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166207 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 07 16:19:30.177935 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166211 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 07 16:19:30.177935 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166215 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 07 16:19:30.177935 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166219 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 07 16:19:30.177935 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166223 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 07 16:19:30.177935 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166228 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 07 16:19:30.177935 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166232 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 07 16:19:30.177935 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166236 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 07 16:19:30.177935 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166240 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 07 16:19:30.177935 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166246 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 07 16:19:30.177935 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166250 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 07 16:19:30.177935 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166254 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 07 16:19:30.177935 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166258 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 07 16:19:30.177935 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166261 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 07 16:19:30.177935 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166268 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 07 16:19:30.177935 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166274 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 07 16:19:30.177935 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166278 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 07 16:19:30.177935 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166284 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 07 16:19:30.177935 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166289 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 07 16:19:30.177935 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166294 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 07 16:19:30.178396 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166298 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 07 16:19:30.178396 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.166303 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 07 16:19:30.178396 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.166950 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 07 16:19:30.178396 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.176755 2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 07 16:19:30.178396 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.176770 2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 07 16:19:30.178789 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.178775 2580 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 07 16:19:30.178817 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.178790 2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 07 16:19:30.178942 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.178932 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 07 16:19:30.178942 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.178941 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 07 16:19:30.179001 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.178945 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 07 16:19:30.179001 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.178949 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 07 16:19:30.179001 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.178952 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 07 16:19:30.179001 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.178961 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 07 16:19:30.179001 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.178964 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 07 16:19:30.179001 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.178968 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 07 16:19:30.179001 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.178971 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 07 16:19:30.179001 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.178977 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 07 16:19:30.179001 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.178979 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 07 16:19:30.179001 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.178982 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 07 16:19:30.179001 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.178985 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 07 16:19:30.179001 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.178988 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 07 16:19:30.179001 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.178991 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 07 16:19:30.179001 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.178993 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 07 16:19:30.179001 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.178996 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 07 16:19:30.179001 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.178999 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 07 16:19:30.179001 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179002 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 07 16:19:30.179001 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179004 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 07 16:19:30.179001 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179007 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 07 16:19:30.179467 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179011 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 07 16:19:30.179467 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179016 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 07 16:19:30.179467 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179019 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 07 16:19:30.179467 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179022 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 07 16:19:30.179467 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179024 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 07 16:19:30.179467 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179027 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 07 16:19:30.179467 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179030 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 07 16:19:30.179467 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179032 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 07 16:19:30.179467 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179035 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 07 16:19:30.179467 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179037 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 07 16:19:30.179467 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179040 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 07 16:19:30.179467 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179043 2580 feature_gate.go:328] unrecognized feature gate: Example2 Sep 07 16:19:30.179467 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179045 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 07 16:19:30.179467 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179048 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 07 16:19:30.179467 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179053 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 07 16:19:30.179467 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179055 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 07 16:19:30.179467 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179058 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 07 16:19:30.179467 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179066 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 07 16:19:30.179467 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179070 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 07 16:19:30.179467 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179074 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 07 16:19:30.179981 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179078 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 07 16:19:30.179981 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179082 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 07 16:19:30.179981 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179086 2580 feature_gate.go:328] unrecognized feature gate: Example Sep 07 16:19:30.179981 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179089 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 07 16:19:30.179981 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179093 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 07 16:19:30.179981 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179099 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 07 16:19:30.179981 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179102 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 07 16:19:30.179981 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179105 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 07 16:19:30.179981 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179108 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 07 16:19:30.179981 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179111 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 07 16:19:30.179981 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179114 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 07 16:19:30.179981 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179117 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 07 16:19:30.179981 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179120 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 07 16:19:30.179981 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179124 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 07 16:19:30.179981 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179126 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 07 16:19:30.179981 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179129 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 07 16:19:30.179981 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179132 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 07 16:19:30.179981 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179135 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 07 16:19:30.179981 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179140 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 07 16:19:30.180470 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179142 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 07 16:19:30.180470 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179145 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 07 16:19:30.180470 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179148 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 07 16:19:30.180470 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179151 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 07 16:19:30.180470 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179153 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 07 16:19:30.180470 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179156 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 07 16:19:30.180470 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179158 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 07 16:19:30.180470 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179161 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 07 16:19:30.180470 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179164 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 07 16:19:30.180470 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179166 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 07 16:19:30.180470 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179169 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 07 16:19:30.180470 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179181 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 07 16:19:30.180470 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179183 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 07 16:19:30.180470 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179186 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 07 16:19:30.180470 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179189 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 07 16:19:30.180470 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179191 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 07 16:19:30.180470 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179194 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 07 16:19:30.180470 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179196 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 07 16:19:30.180470 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179199 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 07 16:19:30.180470 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179202 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 07 16:19:30.181044 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179204 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 07 16:19:30.181044 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179207 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 07 16:19:30.181044 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179209 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 07 16:19:30.181044 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179212 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 07 16:19:30.181044 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179217 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 07 16:19:30.181044 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179219 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 07 16:19:30.181044 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179222 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 07 16:19:30.181044 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179225 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 07 16:19:30.181044 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179228 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 07 16:19:30.181044 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179231 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 07 16:19:30.181044 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179234 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 07 16:19:30.181044 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179236 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 07 16:19:30.181044 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179239 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 07 16:19:30.181044 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179242 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 07 16:19:30.181044 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179244 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 07 16:19:30.181044 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179247 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 07 16:19:30.181044 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179250 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 07 16:19:30.181044 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179255 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 07 16:19:30.181044 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179257 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 07 16:19:30.181512 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.179263 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 07 16:19:30.181512 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179525 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 07 16:19:30.181512 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179531 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 07 16:19:30.181512 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179534 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 07 16:19:30.181512 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179540 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 07 16:19:30.181512 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179549 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 07 16:19:30.181512 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179552 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 07 16:19:30.181512 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179555 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 07 16:19:30.181512 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179558 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 07 16:19:30.181512 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179561 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 07 16:19:30.181512 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179563 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 07 16:19:30.181512 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179566 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 07 16:19:30.181512 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179569 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 07 16:19:30.181512 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179572 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 07 16:19:30.181512 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179575 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 07 16:19:30.181879 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179577 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 07 16:19:30.181879 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179582 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 07 16:19:30.181879 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179585 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 07 16:19:30.181879 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179588 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 07 16:19:30.181879 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179591 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 07 16:19:30.181879 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179594 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 07 16:19:30.181879 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179598 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 07 16:19:30.181879 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179602 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 07 16:19:30.181879 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179605 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 07 16:19:30.181879 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179609 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 07 16:19:30.181879 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179612 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 07 16:19:30.181879 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179615 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 07 16:19:30.181879 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179618 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 07 16:19:30.181879 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179623 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 07 16:19:30.181879 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179626 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 07 16:19:30.181879 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179628 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 07 16:19:30.181879 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179631 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 07 16:19:30.181879 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179635 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 07 16:19:30.182344 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179638 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 07 16:19:30.182344 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179641 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 07 16:19:30.182344 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179644 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 07 16:19:30.182344 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179647 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 07 16:19:30.182344 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179650 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 07 16:19:30.182344 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179658 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 07 16:19:30.182344 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179661 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 07 16:19:30.182344 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179664 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 07 16:19:30.182344 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179669 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 07 16:19:30.182344 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179672 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 07 16:19:30.182344 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179674 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 07 16:19:30.182344 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179677 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 07 16:19:30.182344 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179680 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 07 16:19:30.182344 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179682 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 07 16:19:30.182344 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179685 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 07 16:19:30.182344 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179687 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 07 16:19:30.182344 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179690 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 07 16:19:30.182344 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179692 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 07 16:19:30.182344 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179696 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 07 16:19:30.182344 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179699 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 07 16:19:30.182830 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179702 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 07 16:19:30.182830 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179707 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 07 16:19:30.182830 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179710 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 07 16:19:30.182830 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179712 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 07 16:19:30.182830 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179715 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 07 16:19:30.182830 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179717 2580 feature_gate.go:328] unrecognized feature gate: Example2 Sep 07 16:19:30.182830 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179720 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 07 16:19:30.182830 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179722 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 07 16:19:30.182830 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179725 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 07 16:19:30.182830 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179727 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 07 16:19:30.182830 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179730 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 07 16:19:30.182830 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179733 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 07 16:19:30.182830 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179735 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 07 16:19:30.182830 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179741 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 07 16:19:30.182830 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179743 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 07 16:19:30.182830 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179746 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 07 16:19:30.182830 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179748 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 07 16:19:30.182830 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179751 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 07 16:19:30.182830 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179759 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 07 16:19:30.182830 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179761 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 07 16:19:30.183340 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179764 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 07 16:19:30.183340 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179767 2580 feature_gate.go:328] unrecognized feature gate: Example Sep 07 16:19:30.183340 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179769 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 07 16:19:30.183340 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179772 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 07 16:19:30.183340 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179776 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 07 16:19:30.183340 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179781 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 07 16:19:30.183340 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179784 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 07 16:19:30.183340 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179787 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 07 16:19:30.183340 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179789 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 07 16:19:30.183340 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179792 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 07 16:19:30.183340 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179795 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 07 16:19:30.183340 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179798 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 07 16:19:30.183340 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179800 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 07 16:19:30.183340 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179803 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 07 16:19:30.183340 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179805 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 07 16:19:30.183340 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179808 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 07 16:19:30.183340 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179811 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 07 16:19:30.183340 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179813 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 07 16:19:30.183340 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179818 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 07 16:19:30.183340 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179821 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 07 16:19:30.183826 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179823 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 07 16:19:30.183826 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179826 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 07 16:19:30.183826 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179829 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 07 16:19:30.183826 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179831 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 07 16:19:30.183826 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179834 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 07 16:19:30.183826 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179836 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 07 16:19:30.183826 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:30.179839 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 07 16:19:30.183826 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.179844 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 07 16:19:30.183826 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.180038 2580 server.go:962] "Client rotation is on, will bootstrap in background" Sep 07 16:19:30.183826 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.183132 2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 07 16:19:30.184660 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.184645 2580 server.go:1019] "Starting client certificate rotation" Sep 07 16:19:30.184761 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.184747 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 07 16:19:30.184792 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.184777 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 07 16:19:30.186959 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.186945 2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 07 16:19:30.188157 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.188140 2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 07 16:19:30.203340 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.203315 2580 log.go:25] "Validated CRI v1 runtime API" Sep 07 16:19:30.206480 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.206465 2580 log.go:25] "Validated CRI v1 image API" Sep 07 16:19:30.207461 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.207440 2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 07 16:19:30.209691 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.209676 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 07 16:19:30.212185 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.212168 2580 fs.go:135] Filesystem UUIDs: map[532d7bd4-369a-4c72-bb63-170339cf1db4:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 89f40bfa-c8dc-47bb-8c3e-54cb362eba82:/dev/nvme0n1p4] Sep 07 16:19:30.212236 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.212186 2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:30 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 07 16:19:30.219238 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.219126 2580 manager.go:216] Machine: {Timestamp:2026-09-07 16:19:30.217817661 +0000 UTC m=+0.342170183 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3091233 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2aff5cd45b0df31563737f31ea7e19 SystemUUID:ec2aff5c-d45b-0df3-1563-737f31ea7e19 BootID:c628eadf-93e4-42a3-99ab-64288521d8ed Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:30 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:14:b8:b3:8e:c1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:14:b8:b3:8e:c1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:e6:68:97:f1:54:f2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 07 16:19:30.219238 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.219233 2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 07 16:19:30.219351 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.219312 2580 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 07 16:19:30.222634 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.222605 2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 07 16:19:30.222827 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.222639 2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-242.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 07 16:19:30.224458 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.224445 2580 topology_manager.go:138] "Creating topology manager with none policy" Sep 07 16:19:30.224514 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.224465 2580 container_manager_linux.go:309] "Creating device plugin manager" Sep 07 16:19:30.224514 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.224476 2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 07 16:19:30.225244 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.225232 2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 07 16:19:30.226555 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.226544 2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 07 16:19:30.227841 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.227823 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-p5bwz" Sep 07 16:19:30.230576 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.230562 2580 state_mem.go:36] "Initialized new in-memory state store" Sep 07 16:19:30.230706 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.230695 2580 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 07 16:19:30.230772 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.230735 2580 kubelet.go:486] "Attempting to sync node with API server" Sep 07 16:19:30.230772 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.230749 2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 07 16:19:30.230772 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.230767 2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 07 16:19:30.230909 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.230780 2580 kubelet.go:398] "Adding apiserver pod source" Sep 07 16:19:30.230909 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.230793 2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 07 16:19:30.231239 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.231212 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 07 16:19:30.231347 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.231252 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 07 16:19:30.234514 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.234489 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-p5bwz" Sep 07 16:19:30.237197 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.236559 2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 07 16:19:30.239463 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.239448 2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 07 16:19:30.239525 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.239478 2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 07 16:19:30.240228 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:30.240173 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-242.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 07 16:19:30.240282 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:30.240271 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 07 16:19:30.241458 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.241445 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 07 16:19:30.241502 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.241465 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 07 16:19:30.241502 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.241471 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 07 16:19:30.241502 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.241477 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 07 16:19:30.241502 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.241486 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 07 16:19:30.241502 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.241491 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 07 16:19:30.241502 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.241505 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 07 16:19:30.241668 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.241510 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 07 16:19:30.241668 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.241518 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 07 16:19:30.241668 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.241523 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 07 16:19:30.241668 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.241532 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 07 16:19:30.241668 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.241537 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 07 16:19:30.242267 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.242257 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 07 16:19:30.242267 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.242267 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 07 16:19:30.246248 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.246083 2580 server.go:1268] "Started kubelet" Sep 07 16:19:30.246324 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.246154 2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 07 16:19:30.246382 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.246164 2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 07 16:19:30.246425 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.246410 2580 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 07 16:19:30.247125 ip-10-0-135-242 systemd[1]: Started Kubernetes Kubelet. Sep 07 16:19:30.248329 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.248314 2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 07 16:19:30.249293 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.249277 2580 server.go:310] "Adding debug handlers to kubelet server" Sep 07 16:19:30.251708 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.251689 2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 07 16:19:30.251803 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.251699 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 07 16:19:30.252053 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:30.252010 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-242.ec2.internal\" not found" Sep 07 16:19:30.252290 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.252269 2580 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 07 16:19:30.252290 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.252290 2580 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 07 16:19:30.252440 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.252398 2580 reconstruct.go:97] "Volume reconstruction finished" Sep 07 16:19:30.252440 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.252409 2580 reconciler.go:29] "Reconciler: start to sync state" Sep 07 16:19:30.252700 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.252680 2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 07 16:19:30.253312 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:30.253277 2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 07 16:19:30.253742 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.253728 2580 factory.go:153] Registering CRI-O factory Sep 07 16:19:30.253832 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.253747 2580 factory.go:223] Registration of the crio container factory successfully Sep 07 16:19:30.253832 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.253801 2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 07 16:19:30.253832 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.253810 2580 factory.go:55] Registering systemd factory Sep 07 16:19:30.253832 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.253818 2580 factory.go:223] Registration of the systemd container factory successfully Sep 07 16:19:30.254040 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.253837 2580 factory.go:103] Registering Raw factory Sep 07 16:19:30.254040 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.253850 2580 manager.go:1196] Started watching for new ooms in manager Sep 07 16:19:30.254270 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.254257 2580 manager.go:349] Starting recovery of all containers Sep 07 16:19:30.254429 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.254414 2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 07 16:19:30.257238 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:30.257198 2580 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-135-242.ec2.internal\" not found" node="ip-10-0-135-242.ec2.internal" Sep 07 16:19:30.260557 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.260530 2580 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-135-242.ec2.internal" not found Sep 07 16:19:30.263460 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.263437 2580 manager.go:354] Recovery completed Sep 07 16:19:30.268557 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.268544 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 07 16:19:30.271148 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.271134 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeHasSufficientMemory" Sep 07 16:19:30.271210 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.271163 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeHasNoDiskPressure" Sep 07 16:19:30.271210 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.271173 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeHasSufficientPID" Sep 07 16:19:30.271671 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.271658 2580 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 07 16:19:30.271721 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.271671 2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 07 16:19:30.271721 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.271688 2580 state_mem.go:36] "Initialized new in-memory state store" Sep 07 16:19:30.271721 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.271696 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 07 16:19:30.272466 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.272450 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeHasSufficientMemory" Sep 07 16:19:30.272530 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.272478 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeHasNoDiskPressure" Sep 07 16:19:30.272530 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.272488 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeHasSufficientPID" Sep 07 16:19:30.273948 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.273935 2580 policy_none.go:49] "None policy: Start" Sep 07 16:19:30.273992 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.273951 2580 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 07 16:19:30.273992 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.273962 2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 07 16:19:30.274787 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.274776 2580 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-135-242.ec2.internal" not found Sep 07 16:19:30.276006 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.275995 2580 policy_none.go:47] "Start" Sep 07 16:19:30.308632 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.308611 2580 manager.go:337] "Starting Device Plugin manager" Sep 07 16:19:30.333082 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:30.308663 2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 07 16:19:30.333082 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.308677 2580 server.go:87] "Starting device plugin registration server" Sep 07 16:19:30.333082 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.309059 2580 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 07 16:19:30.333082 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.309072 2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 07 16:19:30.333082 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.309165 2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 07 16:19:30.333082 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.309267 2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 07 16:19:30.333082 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.309276 2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 07 16:19:30.333082 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:30.310068 2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 07 16:19:30.333082 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:30.310116 2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-242.ec2.internal\" not found" Sep 07 16:19:30.333082 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.321217 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 07 16:19:30.333082 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.322614 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 07 16:19:30.333082 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.322653 2580 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 07 16:19:30.333082 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.322677 2580 kubelet.go:2448] "Starting kubelet main sync loop" Sep 07 16:19:30.333082 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:30.322718 2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 07 16:19:30.333082 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.324444 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 07 16:19:30.333082 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.331785 2580 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-135-242.ec2.internal" not found Sep 07 16:19:30.410335 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.410229 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 07 16:19:30.411300 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.411282 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeHasSufficientMemory" Sep 07 16:19:30.411366 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.411313 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeHasNoDiskPressure" Sep 07 16:19:30.411366 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.411323 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeHasSufficientPID" Sep 07 16:19:30.411366 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.411360 2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-242.ec2.internal" Sep 07 16:19:30.419778 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.419765 2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-242.ec2.internal" Sep 07 16:19:30.419825 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:30.419789 2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-242.ec2.internal\": node \"ip-10-0-135-242.ec2.internal\" not found" Sep 07 16:19:30.422782 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.422766 2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-242.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-242.ec2.internal"] Sep 07 16:19:30.422840 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.422833 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 07 16:19:30.423753 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.423737 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeHasSufficientMemory" Sep 07 16:19:30.423804 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.423771 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeHasNoDiskPressure" Sep 07 16:19:30.423804 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.423786 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeHasSufficientPID" Sep 07 16:19:30.426099 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.426086 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 07 16:19:30.426221 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.426206 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-242.ec2.internal" Sep 07 16:19:30.426258 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.426239 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 07 16:19:30.426805 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.426785 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeHasSufficientMemory" Sep 07 16:19:30.426929 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.426817 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeHasNoDiskPressure" Sep 07 16:19:30.426929 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.426828 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeHasSufficientPID" Sep 07 16:19:30.426929 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.426785 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeHasSufficientMemory" Sep 07 16:19:30.426929 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.426886 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeHasNoDiskPressure" Sep 07 16:19:30.426929 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.426918 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeHasSufficientPID" Sep 07 16:19:30.428987 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.428974 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-242.ec2.internal" Sep 07 16:19:30.429070 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.428997 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 07 16:19:30.429703 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.429686 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeHasSufficientMemory" Sep 07 16:19:30.429777 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.429714 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeHasNoDiskPressure" Sep 07 16:19:30.429777 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.429727 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeHasSufficientPID" Sep 07 16:19:30.431961 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:30.431942 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-242.ec2.internal\" not found" Sep 07 16:19:30.444458 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:30.444436 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-242.ec2.internal\" not found" node="ip-10-0-135-242.ec2.internal" Sep 07 16:19:30.447969 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:30.447949 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-242.ec2.internal\" not found" node="ip-10-0-135-242.ec2.internal" Sep 07 16:19:30.532095 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:30.532050 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-242.ec2.internal\" not found" Sep 07 16:19:30.553833 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.553805 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c620202158ca74d37111884761e47e70-config\") pod \"kube-apiserver-proxy-ip-10-0-135-242.ec2.internal\" (UID: \"c620202158ca74d37111884761e47e70\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-242.ec2.internal" Sep 07 16:19:30.553949 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.553844 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8c1647d1d87b8b8396e8fceb50173ea2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-242.ec2.internal\" (UID: \"8c1647d1d87b8b8396e8fceb50173ea2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-242.ec2.internal" Sep 07 16:19:30.553949 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.553867 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c1647d1d87b8b8396e8fceb50173ea2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-242.ec2.internal\" (UID: \"8c1647d1d87b8b8396e8fceb50173ea2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-242.ec2.internal" Sep 07 16:19:30.632307 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:30.632270 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-242.ec2.internal\" not found" Sep 07 16:19:30.654659 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.654627 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8c1647d1d87b8b8396e8fceb50173ea2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-242.ec2.internal\" (UID: \"8c1647d1d87b8b8396e8fceb50173ea2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-242.ec2.internal" Sep 07 16:19:30.654771 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.654667 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c1647d1d87b8b8396e8fceb50173ea2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-242.ec2.internal\" (UID: \"8c1647d1d87b8b8396e8fceb50173ea2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-242.ec2.internal" Sep 07 16:19:30.654771 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.654688 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c620202158ca74d37111884761e47e70-config\") pod \"kube-apiserver-proxy-ip-10-0-135-242.ec2.internal\" (UID: \"c620202158ca74d37111884761e47e70\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-242.ec2.internal" Sep 07 16:19:30.654771 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.654705 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8c1647d1d87b8b8396e8fceb50173ea2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-242.ec2.internal\" (UID: \"8c1647d1d87b8b8396e8fceb50173ea2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-242.ec2.internal" Sep 07 16:19:30.654771 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.654721 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c620202158ca74d37111884761e47e70-config\") pod \"kube-apiserver-proxy-ip-10-0-135-242.ec2.internal\" (UID: \"c620202158ca74d37111884761e47e70\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-242.ec2.internal" Sep 07 16:19:30.654771 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.654721 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c1647d1d87b8b8396e8fceb50173ea2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-242.ec2.internal\" (UID: \"8c1647d1d87b8b8396e8fceb50173ea2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-242.ec2.internal" Sep 07 16:19:30.733101 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:30.733033 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-242.ec2.internal\" not found" Sep 07 16:19:30.748228 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.748188 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-242.ec2.internal" Sep 07 16:19:30.751362 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:30.751347 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-242.ec2.internal" Sep 07 16:19:30.834062 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:30.834022 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-242.ec2.internal\" not found" Sep 07 16:19:30.934587 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:30.934561 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-242.ec2.internal\" not found" Sep 07 16:19:31.035163 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:31.035076 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-242.ec2.internal\" not found" Sep 07 16:19:31.135673 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:31.135642 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-242.ec2.internal\" not found" Sep 07 16:19:31.185081 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:31.185055 2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 07 16:19:31.185537 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:31.185191 2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 07 16:19:31.185537 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:31.185195 2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 07 16:19:31.186903 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:31.186869 2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 07 16:19:31.235873 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:31.235838 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-242.ec2.internal\" not found" Sep 07 16:19:31.239990 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:31.239936 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-06 16:14:31 +0000 UTC" deadline="2028-04-17 21:31:44.59659963 +0000 UTC" Sep 07 16:19:31.240082 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:31.239990 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14117h12m13.356613682s" Sep 07 16:19:31.290957 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:31.290850 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c1647d1d87b8b8396e8fceb50173ea2.slice/crio-85acc791cd9bfcc0c107adda8358937dabeb7d2a247cc9805490e1e4d85ee170 WatchSource:0}: Error finding container 85acc791cd9bfcc0c107adda8358937dabeb7d2a247cc9805490e1e4d85ee170: Status 404 returned error can't find the container with id 85acc791cd9bfcc0c107adda8358937dabeb7d2a247cc9805490e1e4d85ee170 Sep 07 16:19:31.291233 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:31.291207 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc620202158ca74d37111884761e47e70.slice/crio-07970554560e6a65393144d21524b8640858acb007da66777e78179a300e37fb WatchSource:0}: Error finding container 07970554560e6a65393144d21524b8640858acb007da66777e78179a300e37fb: Status 404 returned error can't find the container with id 07970554560e6a65393144d21524b8640858acb007da66777e78179a300e37fb Sep 07 16:19:31.296364 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:31.296346 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 07 16:19:31.326233 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:31.326174 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-242.ec2.internal" event={"ID":"c620202158ca74d37111884761e47e70","Type":"ContainerStarted","Data":"07970554560e6a65393144d21524b8640858acb007da66777e78179a300e37fb"} Sep 07 16:19:31.327605 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:31.327582 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-242.ec2.internal" event={"ID":"8c1647d1d87b8b8396e8fceb50173ea2","Type":"ContainerStarted","Data":"85acc791cd9bfcc0c107adda8358937dabeb7d2a247cc9805490e1e4d85ee170"} Sep 07 16:19:31.336725 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:31.336705 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-242.ec2.internal\" not found" Sep 07 16:19:31.437242 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:31.437206 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-242.ec2.internal\" not found" Sep 07 16:19:31.537785 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:31.537744 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-242.ec2.internal\" not found" Sep 07 16:19:31.638311 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:31.638234 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-242.ec2.internal\" not found" Sep 07 16:19:31.683410 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:31.683382 2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 07 16:19:31.752879 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:31.752846 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-242.ec2.internal" Sep 07 16:19:32.852725 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:32.852592 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 07 16:19:32.854158 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:32.853940 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-242.ec2.internal" Sep 07 16:19:32.861623 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:32.861601 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 07 16:19:33.206512 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.206422 2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 07 16:19:33.321915 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.321883 2580 apiserver.go:52] "Watching apiserver" Sep 07 16:19:33.332695 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.332662 2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 07 16:19:33.333132 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.333099 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-fwmt5","openshift-dns/node-resolver-kcb7q","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-242.ec2.internal","openshift-multus/multus-t4bpj","openshift-multus/network-metrics-daemon-lhv86","openshift-network-operator/iptables-alerter-l9vhv","openshift-ovn-kubernetes/ovnkube-node-rz4mc","kube-system/global-pull-secret-syncer-6n92f","kube-system/konnectivity-agent-hv5bm","kube-system/kube-apiserver-proxy-ip-10-0-135-242.ec2.internal","openshift-image-registry/node-ca-g9wgk","openshift-multus/multus-additional-cni-plugins-7tdcz","openshift-network-diagnostics/network-check-target-8g844","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j6dwj"] Sep 07 16:19:33.335761 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.335741 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:33.338042 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.338005 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 07 16:19:33.338335 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.338316 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-h6rwh\"" Sep 07 16:19:33.338528 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.338514 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 07 16:19:33.338651 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.338639 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 07 16:19:33.338764 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.338749 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 07 16:19:33.338882 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.338870 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 07 16:19:33.339215 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.339186 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 07 16:19:33.339306 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.339213 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8g844" Sep 07 16:19:33.339306 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:33.339289 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8g844" podUID="552ec3ab-d017-450f-85bb-616f65146d2e" Sep 07 16:19:33.340328 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.340311 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j6dwj" Sep 07 16:19:33.342518 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.341840 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 07 16:19:33.342518 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.341915 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 07 16:19:33.342518 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.342225 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 07 16:19:33.342518 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.342506 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-4k8xn\"" Sep 07 16:19:33.342744 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.342575 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 07 16:19:33.342983 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.342966 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-t4bpj" Sep 07 16:19:33.344815 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.344799 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 07 16:19:33.344889 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.344823 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 07 16:19:33.345313 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.345113 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 07 16:19:33.345313 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.345185 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 07 16:19:33.345313 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.345224 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-dkkpw\"" Sep 07 16:19:33.347130 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.347113 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhv86" Sep 07 16:19:33.347222 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:33.347188 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhv86" podUID="599e026d-7338-4319-a1d7-7b714bbb6e58" Sep 07 16:19:33.349755 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.349734 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-l9vhv" Sep 07 16:19:33.351812 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.351794 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 07 16:19:33.351908 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.351829 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 07 16:19:33.352112 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.352093 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-drkrr\"" Sep 07 16:19:33.352196 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.352180 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 07 16:19:33.352720 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.352699 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9039f150-f93a-482e-aa48-46844ac5c7bc-run-ovn\") pod \"ovnkube-node-rz4mc\" (UID: \"9039f150-f93a-482e-aa48-46844ac5c7bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:33.352824 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.352712 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hv5bm" Sep 07 16:19:33.352824 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.352752 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9039f150-f93a-482e-aa48-46844ac5c7bc-node-log\") pod \"ovnkube-node-rz4mc\" (UID: \"9039f150-f93a-482e-aa48-46844ac5c7bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:33.352824 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.352800 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9039f150-f93a-482e-aa48-46844ac5c7bc-host-run-ovn-kubernetes\") pod \"ovnkube-node-rz4mc\" (UID: \"9039f150-f93a-482e-aa48-46844ac5c7bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:33.352824 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.352816 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 07 16:19:33.353004 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.352824 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c1912e17-6316-4af6-9eda-b597816af7e8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j6dwj\" (UID: \"c1912e17-6316-4af6-9eda-b597816af7e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j6dwj" Sep 07 16:19:33.353004 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.352847 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c1912e17-6316-4af6-9eda-b597816af7e8-socket-dir\") pod \"aws-ebs-csi-driver-node-j6dwj\" (UID: \"c1912e17-6316-4af6-9eda-b597816af7e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j6dwj" Sep 07 16:19:33.353004 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.352870 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c1912e17-6316-4af6-9eda-b597816af7e8-etc-selinux\") pod \"aws-ebs-csi-driver-node-j6dwj\" (UID: \"c1912e17-6316-4af6-9eda-b597816af7e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j6dwj" Sep 07 16:19:33.353004 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.352911 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/140ff0c3-223c-4f7a-9142-fc8e999d6449-host-var-lib-cni-multus\") pod \"multus-t4bpj\" (UID: \"140ff0c3-223c-4f7a-9142-fc8e999d6449\") " pod="openshift-multus/multus-t4bpj" Sep 07 16:19:33.353004 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.352935 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9039f150-f93a-482e-aa48-46844ac5c7bc-systemd-units\") pod \"ovnkube-node-rz4mc\" (UID: \"9039f150-f93a-482e-aa48-46844ac5c7bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:33.353004 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.352978 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9039f150-f93a-482e-aa48-46844ac5c7bc-host-cni-netd\") pod \"ovnkube-node-rz4mc\" (UID: \"9039f150-f93a-482e-aa48-46844ac5c7bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:33.353354 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.353017 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9039f150-f93a-482e-aa48-46844ac5c7bc-ovn-node-metrics-cert\") pod \"ovnkube-node-rz4mc\" (UID: \"9039f150-f93a-482e-aa48-46844ac5c7bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:33.353354 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.353082 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckx28\" (UniqueName: \"kubernetes.io/projected/c1912e17-6316-4af6-9eda-b597816af7e8-kube-api-access-ckx28\") pod \"aws-ebs-csi-driver-node-j6dwj\" (UID: \"c1912e17-6316-4af6-9eda-b597816af7e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j6dwj" Sep 07 16:19:33.353354 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.353118 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/140ff0c3-223c-4f7a-9142-fc8e999d6449-host-var-lib-cni-bin\") pod \"multus-t4bpj\" (UID: \"140ff0c3-223c-4f7a-9142-fc8e999d6449\") " pod="openshift-multus/multus-t4bpj" Sep 07 16:19:33.353354 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.353170 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/140ff0c3-223c-4f7a-9142-fc8e999d6449-host-var-lib-kubelet\") pod \"multus-t4bpj\" (UID: \"140ff0c3-223c-4f7a-9142-fc8e999d6449\") " pod="openshift-multus/multus-t4bpj" Sep 07 16:19:33.353354 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.353198 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/140ff0c3-223c-4f7a-9142-fc8e999d6449-multus-daemon-config\") pod \"multus-t4bpj\" (UID: \"140ff0c3-223c-4f7a-9142-fc8e999d6449\") " pod="openshift-multus/multus-t4bpj" Sep 07 16:19:33.353354 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.353224 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/140ff0c3-223c-4f7a-9142-fc8e999d6449-host-run-multus-certs\") pod \"multus-t4bpj\" (UID: \"140ff0c3-223c-4f7a-9142-fc8e999d6449\") " pod="openshift-multus/multus-t4bpj" Sep 07 16:19:33.353354 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.353250 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c1912e17-6316-4af6-9eda-b597816af7e8-device-dir\") pod \"aws-ebs-csi-driver-node-j6dwj\" (UID: \"c1912e17-6316-4af6-9eda-b597816af7e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j6dwj" Sep 07 16:19:33.353354 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.353303 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9039f150-f93a-482e-aa48-46844ac5c7bc-var-lib-openvswitch\") pod \"ovnkube-node-rz4mc\" (UID: \"9039f150-f93a-482e-aa48-46844ac5c7bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:33.353354 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.353349 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9039f150-f93a-482e-aa48-46844ac5c7bc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rz4mc\" (UID: \"9039f150-f93a-482e-aa48-46844ac5c7bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:33.353678 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.353386 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9039f150-f93a-482e-aa48-46844ac5c7bc-ovnkube-config\") pod \"ovnkube-node-rz4mc\" (UID: \"9039f150-f93a-482e-aa48-46844ac5c7bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:33.353678 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.353408 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltwtv\" (UniqueName: \"kubernetes.io/projected/9039f150-f93a-482e-aa48-46844ac5c7bc-kube-api-access-ltwtv\") pod \"ovnkube-node-rz4mc\" (UID: \"9039f150-f93a-482e-aa48-46844ac5c7bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:33.353678 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.353431 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9039f150-f93a-482e-aa48-46844ac5c7bc-host-kubelet\") pod \"ovnkube-node-rz4mc\" (UID: \"9039f150-f93a-482e-aa48-46844ac5c7bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:33.353678 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.353445 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9039f150-f93a-482e-aa48-46844ac5c7bc-ovnkube-script-lib\") pod \"ovnkube-node-rz4mc\" (UID: \"9039f150-f93a-482e-aa48-46844ac5c7bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:33.353678 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.353459 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcgl7\" (UniqueName: \"kubernetes.io/projected/552ec3ab-d017-450f-85bb-616f65146d2e-kube-api-access-kcgl7\") pod \"network-check-target-8g844\" (UID: \"552ec3ab-d017-450f-85bb-616f65146d2e\") " pod="openshift-network-diagnostics/network-check-target-8g844" Sep 07 16:19:33.353678 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.353481 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c1912e17-6316-4af6-9eda-b597816af7e8-registration-dir\") pod \"aws-ebs-csi-driver-node-j6dwj\" (UID: \"c1912e17-6316-4af6-9eda-b597816af7e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j6dwj" Sep 07 16:19:33.353678 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.353503 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/140ff0c3-223c-4f7a-9142-fc8e999d6449-etc-kubernetes\") pod \"multus-t4bpj\" (UID: \"140ff0c3-223c-4f7a-9142-fc8e999d6449\") " pod="openshift-multus/multus-t4bpj" Sep 07 16:19:33.353678 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.353523 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxt2d\" (UniqueName: \"kubernetes.io/projected/140ff0c3-223c-4f7a-9142-fc8e999d6449-kube-api-access-fxt2d\") pod \"multus-t4bpj\" (UID: \"140ff0c3-223c-4f7a-9142-fc8e999d6449\") " pod="openshift-multus/multus-t4bpj" Sep 07 16:19:33.353678 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.353545 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/140ff0c3-223c-4f7a-9142-fc8e999d6449-cnibin\") pod \"multus-t4bpj\" (UID: \"140ff0c3-223c-4f7a-9142-fc8e999d6449\") " pod="openshift-multus/multus-t4bpj" Sep 07 16:19:33.353678 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.353574 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/140ff0c3-223c-4f7a-9142-fc8e999d6449-cni-binary-copy\") pod \"multus-t4bpj\" (UID: \"140ff0c3-223c-4f7a-9142-fc8e999d6449\") " pod="openshift-multus/multus-t4bpj" Sep 07 16:19:33.353678 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.353593 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/140ff0c3-223c-4f7a-9142-fc8e999d6449-host-run-k8s-cni-cncf-io\") pod \"multus-t4bpj\" (UID: \"140ff0c3-223c-4f7a-9142-fc8e999d6449\") " pod="openshift-multus/multus-t4bpj" Sep 07 16:19:33.353678 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.353608 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/140ff0c3-223c-4f7a-9142-fc8e999d6449-host-run-netns\") pod \"multus-t4bpj\" (UID: \"140ff0c3-223c-4f7a-9142-fc8e999d6449\") " pod="openshift-multus/multus-t4bpj" Sep 07 16:19:33.353678 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.353628 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/140ff0c3-223c-4f7a-9142-fc8e999d6449-hostroot\") pod \"multus-t4bpj\" (UID: \"140ff0c3-223c-4f7a-9142-fc8e999d6449\") " pod="openshift-multus/multus-t4bpj" Sep 07 16:19:33.353678 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.353651 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/140ff0c3-223c-4f7a-9142-fc8e999d6449-multus-conf-dir\") pod \"multus-t4bpj\" (UID: \"140ff0c3-223c-4f7a-9142-fc8e999d6449\") " pod="openshift-multus/multus-t4bpj" Sep 07 16:19:33.353678 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.353668 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9039f150-f93a-482e-aa48-46844ac5c7bc-host-slash\") pod \"ovnkube-node-rz4mc\" (UID: \"9039f150-f93a-482e-aa48-46844ac5c7bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:33.353678 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.353681 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9039f150-f93a-482e-aa48-46844ac5c7bc-host-run-netns\") pod \"ovnkube-node-rz4mc\" (UID: \"9039f150-f93a-482e-aa48-46844ac5c7bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:33.354164 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.353695 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9039f150-f93a-482e-aa48-46844ac5c7bc-log-socket\") pod \"ovnkube-node-rz4mc\" (UID: \"9039f150-f93a-482e-aa48-46844ac5c7bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:33.354164 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.353714 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9039f150-f93a-482e-aa48-46844ac5c7bc-env-overrides\") pod \"ovnkube-node-rz4mc\" (UID: \"9039f150-f93a-482e-aa48-46844ac5c7bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:33.354164 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.353740 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/140ff0c3-223c-4f7a-9142-fc8e999d6449-system-cni-dir\") pod \"multus-t4bpj\" (UID: \"140ff0c3-223c-4f7a-9142-fc8e999d6449\") " pod="openshift-multus/multus-t4bpj" Sep 07 16:19:33.354164 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.353808 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9039f150-f93a-482e-aa48-46844ac5c7bc-run-systemd\") pod \"ovnkube-node-rz4mc\" (UID: \"9039f150-f93a-482e-aa48-46844ac5c7bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:33.354164 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.353848 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9039f150-f93a-482e-aa48-46844ac5c7bc-etc-openvswitch\") pod \"ovnkube-node-rz4mc\" (UID: \"9039f150-f93a-482e-aa48-46844ac5c7bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:33.354164 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.353871 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9039f150-f93a-482e-aa48-46844ac5c7bc-host-cni-bin\") pod \"ovnkube-node-rz4mc\" (UID: \"9039f150-f93a-482e-aa48-46844ac5c7bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:33.354164 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.353898 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c1912e17-6316-4af6-9eda-b597816af7e8-sys-fs\") pod \"aws-ebs-csi-driver-node-j6dwj\" (UID: \"c1912e17-6316-4af6-9eda-b597816af7e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j6dwj" Sep 07 16:19:33.354164 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.353940 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/140ff0c3-223c-4f7a-9142-fc8e999d6449-multus-cni-dir\") pod \"multus-t4bpj\" (UID: \"140ff0c3-223c-4f7a-9142-fc8e999d6449\") " pod="openshift-multus/multus-t4bpj" Sep 07 16:19:33.354164 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.353966 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9039f150-f93a-482e-aa48-46844ac5c7bc-run-openvswitch\") pod \"ovnkube-node-rz4mc\" (UID: \"9039f150-f93a-482e-aa48-46844ac5c7bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:33.354164 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.353989 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/140ff0c3-223c-4f7a-9142-fc8e999d6449-multus-socket-dir-parent\") pod \"multus-t4bpj\" (UID: \"140ff0c3-223c-4f7a-9142-fc8e999d6449\") " pod="openshift-multus/multus-t4bpj" Sep 07 16:19:33.354693 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.354675 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 07 16:19:33.354754 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.354737 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-nvr2w\"" Sep 07 16:19:33.354956 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.354940 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 07 16:19:33.355277 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.355257 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fwmt5" Sep 07 16:19:33.356991 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.356971 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 07 16:19:33.357307 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.357287 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 07 16:19:33.357446 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.357428 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kcb7q" Sep 07 16:19:33.357869 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.357851 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rt544\"" Sep 07 16:19:33.359451 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.359430 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 07 16:19:33.359701 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.359678 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6n92f" Sep 07 16:19:33.360688 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:33.360650 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6n92f" podUID="4f4aaec6-7ab4-48ab-a0e1-4d26977f0315" Sep 07 16:19:33.360688 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.360681 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 07 16:19:33.364112 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.361281 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mkxmh\"" Sep 07 16:19:33.364912 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.364891 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-g9wgk" Sep 07 16:19:33.366925 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.366903 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 07 16:19:33.367168 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.367151 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6z7w5\"" Sep 07 16:19:33.367262 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.367198 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 07 16:19:33.367262 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.367154 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 07 16:19:33.368580 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.368561 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7tdcz" Sep 07 16:19:33.370314 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.370295 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 07 16:19:33.370400 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.370357 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 07 16:19:33.370454 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.370298 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-t6tcq\"" Sep 07 16:19:33.374309 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.374290 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-6qk5w" Sep 07 16:19:33.382197 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.382164 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-6qk5w" Sep 07 16:19:33.442663 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.442635 2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 07 16:19:33.454633 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.454604 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c1912e17-6316-4af6-9eda-b597816af7e8-sys-fs\") pod \"aws-ebs-csi-driver-node-j6dwj\" (UID: \"c1912e17-6316-4af6-9eda-b597816af7e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j6dwj" Sep 07 16:19:33.454810 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.454653 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/53ae957e-ec96-4d71-bca5-908c965ec61a-etc-sysctl-conf\") pod \"tuned-fwmt5\" (UID: \"53ae957e-ec96-4d71-bca5-908c965ec61a\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmt5" Sep 07 16:19:33.454810 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.454676 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/53ae957e-ec96-4d71-bca5-908c965ec61a-host\") pod \"tuned-fwmt5\" (UID: \"53ae957e-ec96-4d71-bca5-908c965ec61a\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmt5" Sep 07 16:19:33.454810 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.454700 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2e77c59f-ef45-4fb8-99be-5fdccb594e0c-cnibin\") pod \"multus-additional-cni-plugins-7tdcz\" (UID: \"2e77c59f-ef45-4fb8-99be-5fdccb594e0c\") " pod="openshift-multus/multus-additional-cni-plugins-7tdcz" Sep 07 16:19:33.454810 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.454730 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9039f150-f93a-482e-aa48-46844ac5c7bc-run-openvswitch\") pod \"ovnkube-node-rz4mc\" (UID: \"9039f150-f93a-482e-aa48-46844ac5c7bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:33.454810 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.454735 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c1912e17-6316-4af6-9eda-b597816af7e8-sys-fs\") pod \"aws-ebs-csi-driver-node-j6dwj\" (UID: \"c1912e17-6316-4af6-9eda-b597816af7e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j6dwj" Sep 07 16:19:33.454810 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.454768 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9039f150-f93a-482e-aa48-46844ac5c7bc-run-openvswitch\") pod \"ovnkube-node-rz4mc\" (UID: \"9039f150-f93a-482e-aa48-46844ac5c7bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:33.454810 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.454757 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/140ff0c3-223c-4f7a-9142-fc8e999d6449-multus-socket-dir-parent\") pod \"multus-t4bpj\" (UID: \"140ff0c3-223c-4f7a-9142-fc8e999d6449\") " pod="openshift-multus/multus-t4bpj" Sep 07 16:19:33.454810 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.454797 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/140ff0c3-223c-4f7a-9142-fc8e999d6449-multus-socket-dir-parent\") pod \"multus-t4bpj\" (UID: \"140ff0c3-223c-4f7a-9142-fc8e999d6449\") " pod="openshift-multus/multus-t4bpj" Sep 07 16:19:33.455214 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.454812 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/53ae957e-ec96-4d71-bca5-908c965ec61a-etc-tuned\") pod \"tuned-fwmt5\" (UID: \"53ae957e-ec96-4d71-bca5-908c965ec61a\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmt5" Sep 07 16:19:33.455214 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.454897 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8c534a46-aee8-4f33-a447-c049fb3d3f04-serviceca\") pod \"node-ca-g9wgk\" (UID: \"8c534a46-aee8-4f33-a447-c049fb3d3f04\") " pod="openshift-image-registry/node-ca-g9wgk" Sep 07 16:19:33.455214 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.454927 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hm2q\" (UniqueName: \"kubernetes.io/projected/2e77c59f-ef45-4fb8-99be-5fdccb594e0c-kube-api-access-2hm2q\") pod \"multus-additional-cni-plugins-7tdcz\" (UID: \"2e77c59f-ef45-4fb8-99be-5fdccb594e0c\") " pod="openshift-multus/multus-additional-cni-plugins-7tdcz" Sep 07 16:19:33.455214 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.454956 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9039f150-f93a-482e-aa48-46844ac5c7bc-run-ovn\") pod \"ovnkube-node-rz4mc\" (UID: \"9039f150-f93a-482e-aa48-46844ac5c7bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:33.455214 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.454986 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9039f150-f93a-482e-aa48-46844ac5c7bc-host-run-ovn-kubernetes\") pod \"ovnkube-node-rz4mc\" (UID: \"9039f150-f93a-482e-aa48-46844ac5c7bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:33.455214 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.455002 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c1912e17-6316-4af6-9eda-b597816af7e8-socket-dir\") pod \"aws-ebs-csi-driver-node-j6dwj\" (UID: \"c1912e17-6316-4af6-9eda-b597816af7e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j6dwj" Sep 07 16:19:33.455214 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.455016 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c1912e17-6316-4af6-9eda-b597816af7e8-etc-selinux\") pod \"aws-ebs-csi-driver-node-j6dwj\" (UID: \"c1912e17-6316-4af6-9eda-b597816af7e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j6dwj" Sep 07 16:19:33.455214 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.455053 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/140ff0c3-223c-4f7a-9142-fc8e999d6449-host-var-lib-cni-multus\") pod \"multus-t4bpj\" (UID: \"140ff0c3-223c-4f7a-9142-fc8e999d6449\") " pod="openshift-multus/multus-t4bpj" Sep 07 16:19:33.455214 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.455068 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9039f150-f93a-482e-aa48-46844ac5c7bc-host-run-ovn-kubernetes\") pod \"ovnkube-node-rz4mc\" (UID: \"9039f150-f93a-482e-aa48-46844ac5c7bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:33.455214 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.455073 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/140ff0c3-223c-4f7a-9142-fc8e999d6449-host-var-lib-kubelet\") pod \"multus-t4bpj\" (UID: \"140ff0c3-223c-4f7a-9142-fc8e999d6449\") " pod="openshift-multus/multus-t4bpj" Sep 07 16:19:33.455214 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.455107 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9dd316ae-6ec2-4359-887d-7e2665504ce4-host-slash\") pod \"iptables-alerter-l9vhv\" (UID: \"9dd316ae-6ec2-4359-887d-7e2665504ce4\") " pod="openshift-network-operator/iptables-alerter-l9vhv" Sep 07 16:19:33.455214 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.455109 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/140ff0c3-223c-4f7a-9142-fc8e999d6449-host-var-lib-kubelet\") pod \"multus-t4bpj\" (UID: \"140ff0c3-223c-4f7a-9142-fc8e999d6449\") " pod="openshift-multus/multus-t4bpj" Sep 07 16:19:33.455214 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.455127 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9039f150-f93a-482e-aa48-46844ac5c7bc-host-cni-netd\") pod \"ovnkube-node-rz4mc\" (UID: \"9039f150-f93a-482e-aa48-46844ac5c7bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:33.455214 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.455144 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9039f150-f93a-482e-aa48-46844ac5c7bc-ovn-node-metrics-cert\") pod \"ovnkube-node-rz4mc\" (UID: \"9039f150-f93a-482e-aa48-46844ac5c7bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:33.455214 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.455155 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/140ff0c3-223c-4f7a-9142-fc8e999d6449-host-var-lib-cni-multus\") pod \"multus-t4bpj\" (UID: \"140ff0c3-223c-4f7a-9142-fc8e999d6449\") " pod="openshift-multus/multus-t4bpj" Sep 07 16:19:33.455214 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.455158 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckx28\" (UniqueName: \"kubernetes.io/projected/c1912e17-6316-4af6-9eda-b597816af7e8-kube-api-access-ckx28\") pod \"aws-ebs-csi-driver-node-j6dwj\" (UID: \"c1912e17-6316-4af6-9eda-b597816af7e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j6dwj" Sep 07 16:19:33.455214 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.455174 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/140ff0c3-223c-4f7a-9142-fc8e999d6449-host-run-multus-certs\") pod \"multus-t4bpj\" (UID: \"140ff0c3-223c-4f7a-9142-fc8e999d6449\") " pod="openshift-multus/multus-t4bpj" Sep 07 16:19:33.455928 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.455185 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9039f150-f93a-482e-aa48-46844ac5c7bc-run-ovn\") pod \"ovnkube-node-rz4mc\" (UID: \"9039f150-f93a-482e-aa48-46844ac5c7bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:33.455928 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.455184 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c1912e17-6316-4af6-9eda-b597816af7e8-etc-selinux\") pod \"aws-ebs-csi-driver-node-j6dwj\" (UID: \"c1912e17-6316-4af6-9eda-b597816af7e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j6dwj" Sep 07 16:19:33.455928 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.455222 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9039f150-f93a-482e-aa48-46844ac5c7bc-host-cni-netd\") pod \"ovnkube-node-rz4mc\" (UID: \"9039f150-f93a-482e-aa48-46844ac5c7bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:33.455928 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.455372 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c1912e17-6316-4af6-9eda-b597816af7e8-socket-dir\") pod \"aws-ebs-csi-driver-node-j6dwj\" (UID: \"c1912e17-6316-4af6-9eda-b597816af7e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j6dwj" Sep 07 16:19:33.455928 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.455420 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/140ff0c3-223c-4f7a-9142-fc8e999d6449-host-run-multus-certs\") pod \"multus-t4bpj\" (UID: \"140ff0c3-223c-4f7a-9142-fc8e999d6449\") " pod="openshift-multus/multus-t4bpj" Sep 07 16:19:33.455928 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.455192 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/553ed3c9-3691-4532-8a44-702ea20b17a7-hosts-file\") pod \"node-resolver-kcb7q\" (UID: \"553ed3c9-3691-4532-8a44-702ea20b17a7\") " pod="openshift-dns/node-resolver-kcb7q" Sep 07 16:19:33.455928 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.455456 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/53ae957e-ec96-4d71-bca5-908c965ec61a-etc-sysctl-d\") pod \"tuned-fwmt5\" (UID: \"53ae957e-ec96-4d71-bca5-908c965ec61a\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmt5" Sep 07 16:19:33.455928 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.455478 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/53ae957e-ec96-4d71-bca5-908c965ec61a-etc-systemd\") pod \"tuned-fwmt5\" (UID: \"53ae957e-ec96-4d71-bca5-908c965ec61a\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmt5" Sep 07 16:19:33.455928 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.455495 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn5wf\" (UniqueName: \"kubernetes.io/projected/8c534a46-aee8-4f33-a447-c049fb3d3f04-kube-api-access-sn5wf\") pod \"node-ca-g9wgk\" (UID: \"8c534a46-aee8-4f33-a447-c049fb3d3f04\") " pod="openshift-image-registry/node-ca-g9wgk" Sep 07 16:19:33.455928 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.455538 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c1912e17-6316-4af6-9eda-b597816af7e8-device-dir\") pod \"aws-ebs-csi-driver-node-j6dwj\" (UID: \"c1912e17-6316-4af6-9eda-b597816af7e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j6dwj" Sep 07 16:19:33.455928 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.455554 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjzmg\" (UniqueName: \"kubernetes.io/projected/9dd316ae-6ec2-4359-887d-7e2665504ce4-kube-api-access-xjzmg\") pod \"iptables-alerter-l9vhv\" (UID: \"9dd316ae-6ec2-4359-887d-7e2665504ce4\") " pod="openshift-network-operator/iptables-alerter-l9vhv" Sep 07 16:19:33.455928 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.455581 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4013a04f-2b59-44fa-af92-aece71a27fc8-konnectivity-ca\") pod \"konnectivity-agent-hv5bm\" (UID: \"4013a04f-2b59-44fa-af92-aece71a27fc8\") " pod="kube-system/konnectivity-agent-hv5bm" Sep 07 16:19:33.455928 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.455668 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c1912e17-6316-4af6-9eda-b597816af7e8-device-dir\") pod \"aws-ebs-csi-driver-node-j6dwj\" (UID: \"c1912e17-6316-4af6-9eda-b597816af7e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j6dwj" Sep 07 16:19:33.455928 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.455694 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/53ae957e-ec96-4d71-bca5-908c965ec61a-lib-modules\") pod \"tuned-fwmt5\" (UID: \"53ae957e-ec96-4d71-bca5-908c965ec61a\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmt5" Sep 07 16:19:33.455928 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.455722 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2e77c59f-ef45-4fb8-99be-5fdccb594e0c-system-cni-dir\") pod \"multus-additional-cni-plugins-7tdcz\" (UID: \"2e77c59f-ef45-4fb8-99be-5fdccb594e0c\") " pod="openshift-multus/multus-additional-cni-plugins-7tdcz" Sep 07 16:19:33.455928 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.455740 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2e77c59f-ef45-4fb8-99be-5fdccb594e0c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7tdcz\" (UID: \"2e77c59f-ef45-4fb8-99be-5fdccb594e0c\") " pod="openshift-multus/multus-additional-cni-plugins-7tdcz" Sep 07 16:19:33.455928 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.455793 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9039f150-f93a-482e-aa48-46844ac5c7bc-ovnkube-script-lib\") pod \"ovnkube-node-rz4mc\" (UID: \"9039f150-f93a-482e-aa48-46844ac5c7bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:33.456665 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.455826 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/53ae957e-ec96-4d71-bca5-908c965ec61a-etc-modprobe-d\") pod \"tuned-fwmt5\" (UID: \"53ae957e-ec96-4d71-bca5-908c965ec61a\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmt5" Sep 07 16:19:33.456665 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.455875 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/140ff0c3-223c-4f7a-9142-fc8e999d6449-cni-binary-copy\") pod \"multus-t4bpj\" (UID: \"140ff0c3-223c-4f7a-9142-fc8e999d6449\") " pod="openshift-multus/multus-t4bpj" Sep 07 16:19:33.456665 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.455895 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/140ff0c3-223c-4f7a-9142-fc8e999d6449-host-run-k8s-cni-cncf-io\") pod \"multus-t4bpj\" (UID: \"140ff0c3-223c-4f7a-9142-fc8e999d6449\") " pod="openshift-multus/multus-t4bpj" Sep 07 16:19:33.456665 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.455919 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/140ff0c3-223c-4f7a-9142-fc8e999d6449-multus-conf-dir\") pod \"multus-t4bpj\" (UID: \"140ff0c3-223c-4f7a-9142-fc8e999d6449\") " pod="openshift-multus/multus-t4bpj" Sep 07 16:19:33.456665 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.455944 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/53ae957e-ec96-4d71-bca5-908c965ec61a-etc-kubernetes\") pod \"tuned-fwmt5\" (UID: \"53ae957e-ec96-4d71-bca5-908c965ec61a\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmt5" Sep 07 16:19:33.456665 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.455976 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9039f150-f93a-482e-aa48-46844ac5c7bc-env-overrides\") pod \"ovnkube-node-rz4mc\" (UID: \"9039f150-f93a-482e-aa48-46844ac5c7bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:33.456665 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.455988 2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 07 16:19:33.456665 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.456001 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/140ff0c3-223c-4f7a-9142-fc8e999d6449-system-cni-dir\") pod \"multus-t4bpj\" (UID: \"140ff0c3-223c-4f7a-9142-fc8e999d6449\") " pod="openshift-multus/multus-t4bpj" Sep 07 16:19:33.456665 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.456063 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2e77c59f-ef45-4fb8-99be-5fdccb594e0c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7tdcz\" (UID: \"2e77c59f-ef45-4fb8-99be-5fdccb594e0c\") " pod="openshift-multus/multus-additional-cni-plugins-7tdcz" Sep 07 16:19:33.456665 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.456090 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9039f150-f93a-482e-aa48-46844ac5c7bc-run-systemd\") pod \"ovnkube-node-rz4mc\" (UID: \"9039f150-f93a-482e-aa48-46844ac5c7bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:33.456665 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.456112 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9039f150-f93a-482e-aa48-46844ac5c7bc-etc-openvswitch\") pod \"ovnkube-node-rz4mc\" (UID: \"9039f150-f93a-482e-aa48-46844ac5c7bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:33.456665 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.456153 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9039f150-f93a-482e-aa48-46844ac5c7bc-host-cni-bin\") pod \"ovnkube-node-rz4mc\" (UID: \"9039f150-f93a-482e-aa48-46844ac5c7bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:33.456665 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.456177 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/140ff0c3-223c-4f7a-9142-fc8e999d6449-multus-cni-dir\") pod \"multus-t4bpj\" (UID: \"140ff0c3-223c-4f7a-9142-fc8e999d6449\") " pod="openshift-multus/multus-t4bpj" Sep 07 16:19:33.456665 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.456225 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8c534a46-aee8-4f33-a447-c049fb3d3f04-host\") pod \"node-ca-g9wgk\" (UID: \"8c534a46-aee8-4f33-a447-c049fb3d3f04\") " pod="openshift-image-registry/node-ca-g9wgk" Sep 07 16:19:33.456665 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.456258 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/599e026d-7338-4319-a1d7-7b714bbb6e58-metrics-certs\") pod \"network-metrics-daemon-lhv86\" (UID: \"599e026d-7338-4319-a1d7-7b714bbb6e58\") " pod="openshift-multus/network-metrics-daemon-lhv86" Sep 07 16:19:33.456665 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.456298 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9039f150-f93a-482e-aa48-46844ac5c7bc-node-log\") pod \"ovnkube-node-rz4mc\" (UID: \"9039f150-f93a-482e-aa48-46844ac5c7bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:33.456665 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.456364 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c1912e17-6316-4af6-9eda-b597816af7e8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j6dwj\" (UID: \"c1912e17-6316-4af6-9eda-b597816af7e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j6dwj" Sep 07 16:19:33.456665 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.456388 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/140ff0c3-223c-4f7a-9142-fc8e999d6449-multus-daemon-config\") pod \"multus-t4bpj\" (UID: \"140ff0c3-223c-4f7a-9142-fc8e999d6449\") " pod="openshift-multus/multus-t4bpj" Sep 07 16:19:33.457379 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.456488 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/140ff0c3-223c-4f7a-9142-fc8e999d6449-etc-kubernetes\") pod \"multus-t4bpj\" (UID: \"140ff0c3-223c-4f7a-9142-fc8e999d6449\") " pod="openshift-multus/multus-t4bpj" Sep 07 16:19:33.457379 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.456513 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxt2d\" (UniqueName: \"kubernetes.io/projected/140ff0c3-223c-4f7a-9142-fc8e999d6449-kube-api-access-fxt2d\") pod \"multus-t4bpj\" (UID: \"140ff0c3-223c-4f7a-9142-fc8e999d6449\") " pod="openshift-multus/multus-t4bpj" Sep 07 16:19:33.457379 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.456536 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwskj\" (UniqueName: \"kubernetes.io/projected/599e026d-7338-4319-a1d7-7b714bbb6e58-kube-api-access-qwskj\") pod \"network-metrics-daemon-lhv86\" (UID: \"599e026d-7338-4319-a1d7-7b714bbb6e58\") " pod="openshift-multus/network-metrics-daemon-lhv86" Sep 07 16:19:33.457379 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.456574 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9039f150-f93a-482e-aa48-46844ac5c7bc-systemd-units\") pod \"ovnkube-node-rz4mc\" (UID: \"9039f150-f93a-482e-aa48-46844ac5c7bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:33.457379 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.456605 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/140ff0c3-223c-4f7a-9142-fc8e999d6449-cni-binary-copy\") pod \"multus-t4bpj\" (UID: \"140ff0c3-223c-4f7a-9142-fc8e999d6449\") " pod="openshift-multus/multus-t4bpj" Sep 07 16:19:33.457379 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.456607 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9039f150-f93a-482e-aa48-46844ac5c7bc-ovnkube-script-lib\") pod \"ovnkube-node-rz4mc\" (UID: \"9039f150-f93a-482e-aa48-46844ac5c7bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:33.457379 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.456608 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/140ff0c3-223c-4f7a-9142-fc8e999d6449-host-var-lib-cni-bin\") pod \"multus-t4bpj\" (UID: \"140ff0c3-223c-4f7a-9142-fc8e999d6449\") " pod="openshift-multus/multus-t4bpj" Sep 07 16:19:33.457379 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.456662 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/140ff0c3-223c-4f7a-9142-fc8e999d6449-host-var-lib-cni-bin\") pod \"multus-t4bpj\" (UID: \"140ff0c3-223c-4f7a-9142-fc8e999d6449\") " pod="openshift-multus/multus-t4bpj" Sep 07 16:19:33.457379 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.456673 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9dd316ae-6ec2-4359-887d-7e2665504ce4-iptables-alerter-script\") pod \"iptables-alerter-l9vhv\" (UID: \"9dd316ae-6ec2-4359-887d-7e2665504ce4\") " pod="openshift-network-operator/iptables-alerter-l9vhv" Sep 07 16:19:33.457379 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.456704 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/53ae957e-ec96-4d71-bca5-908c965ec61a-run\") pod \"tuned-fwmt5\" (UID: \"53ae957e-ec96-4d71-bca5-908c965ec61a\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmt5" Sep 07 16:19:33.457379 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.456767 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9039f150-f93a-482e-aa48-46844ac5c7bc-systemd-units\") pod \"ovnkube-node-rz4mc\" (UID: \"9039f150-f93a-482e-aa48-46844ac5c7bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:33.457379 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.456862 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9039f150-f93a-482e-aa48-46844ac5c7bc-run-systemd\") pod \"ovnkube-node-rz4mc\" (UID: \"9039f150-f93a-482e-aa48-46844ac5c7bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:33.457379 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.456955 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/140ff0c3-223c-4f7a-9142-fc8e999d6449-system-cni-dir\") pod \"multus-t4bpj\" (UID: \"140ff0c3-223c-4f7a-9142-fc8e999d6449\") " pod="openshift-multus/multus-t4bpj" Sep 07 16:19:33.457379 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.456989 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9039f150-f93a-482e-aa48-46844ac5c7bc-var-lib-openvswitch\") pod \"ovnkube-node-rz4mc\" (UID: \"9039f150-f93a-482e-aa48-46844ac5c7bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:33.457379 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.457012 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9039f150-f93a-482e-aa48-46844ac5c7bc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rz4mc\" (UID: \"9039f150-f93a-482e-aa48-46844ac5c7bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:33.457379 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.457054 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9039f150-f93a-482e-aa48-46844ac5c7bc-ovnkube-config\") pod \"ovnkube-node-rz4mc\" (UID: \"9039f150-f93a-482e-aa48-46844ac5c7bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:33.457379 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.457081 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltwtv\" (UniqueName: \"kubernetes.io/projected/9039f150-f93a-482e-aa48-46844ac5c7bc-kube-api-access-ltwtv\") pod \"ovnkube-node-rz4mc\" (UID: \"9039f150-f93a-482e-aa48-46844ac5c7bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:33.458260 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.457106 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c1912e17-6316-4af6-9eda-b597816af7e8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j6dwj\" (UID: \"c1912e17-6316-4af6-9eda-b597816af7e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j6dwj" Sep 07 16:19:33.458260 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.457108 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr5zh\" (UniqueName: \"kubernetes.io/projected/553ed3c9-3691-4532-8a44-702ea20b17a7-kube-api-access-zr5zh\") pod \"node-resolver-kcb7q\" (UID: \"553ed3c9-3691-4532-8a44-702ea20b17a7\") " pod="openshift-dns/node-resolver-kcb7q" Sep 07 16:19:33.458260 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.457147 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/53ae957e-ec96-4d71-bca5-908c965ec61a-etc-sysconfig\") pod \"tuned-fwmt5\" (UID: \"53ae957e-ec96-4d71-bca5-908c965ec61a\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmt5" Sep 07 16:19:33.458260 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.457152 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/140ff0c3-223c-4f7a-9142-fc8e999d6449-etc-kubernetes\") pod \"multus-t4bpj\" (UID: \"140ff0c3-223c-4f7a-9142-fc8e999d6449\") " pod="openshift-multus/multus-t4bpj" Sep 07 16:19:33.458260 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.457150 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9039f150-f93a-482e-aa48-46844ac5c7bc-env-overrides\") pod \"ovnkube-node-rz4mc\" (UID: \"9039f150-f93a-482e-aa48-46844ac5c7bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:33.458260 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.457168 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/140ff0c3-223c-4f7a-9142-fc8e999d6449-multus-daemon-config\") pod \"multus-t4bpj\" (UID: \"140ff0c3-223c-4f7a-9142-fc8e999d6449\") " pod="openshift-multus/multus-t4bpj" Sep 07 16:19:33.458260 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.457185 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9039f150-f93a-482e-aa48-46844ac5c7bc-node-log\") pod \"ovnkube-node-rz4mc\" (UID: \"9039f150-f93a-482e-aa48-46844ac5c7bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:33.458260 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.457202 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9039f150-f93a-482e-aa48-46844ac5c7bc-etc-openvswitch\") pod \"ovnkube-node-rz4mc\" (UID: \"9039f150-f93a-482e-aa48-46844ac5c7bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:33.458260 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.457202 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/53ae957e-ec96-4d71-bca5-908c965ec61a-sys\") pod \"tuned-fwmt5\" (UID: \"53ae957e-ec96-4d71-bca5-908c965ec61a\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmt5" Sep 07 16:19:33.458260 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.457217 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9039f150-f93a-482e-aa48-46844ac5c7bc-host-cni-bin\") pod \"ovnkube-node-rz4mc\" (UID: \"9039f150-f93a-482e-aa48-46844ac5c7bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:33.458260 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.457229 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/53ae957e-ec96-4d71-bca5-908c965ec61a-var-lib-kubelet\") pod \"tuned-fwmt5\" (UID: \"53ae957e-ec96-4d71-bca5-908c965ec61a\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmt5" Sep 07 16:19:33.458260 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.457225 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9039f150-f93a-482e-aa48-46844ac5c7bc-var-lib-openvswitch\") pod \"ovnkube-node-rz4mc\" (UID: \"9039f150-f93a-482e-aa48-46844ac5c7bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:33.458260 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.457256 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/140ff0c3-223c-4f7a-9142-fc8e999d6449-host-run-k8s-cni-cncf-io\") pod \"multus-t4bpj\" (UID: \"140ff0c3-223c-4f7a-9142-fc8e999d6449\") " pod="openshift-multus/multus-t4bpj" Sep 07 16:19:33.458260 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.457274 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9039f150-f93a-482e-aa48-46844ac5c7bc-host-kubelet\") pod \"ovnkube-node-rz4mc\" (UID: \"9039f150-f93a-482e-aa48-46844ac5c7bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:33.458260 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.457279 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9039f150-f93a-482e-aa48-46844ac5c7bc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rz4mc\" (UID: \"9039f150-f93a-482e-aa48-46844ac5c7bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:33.458260 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.457285 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/140ff0c3-223c-4f7a-9142-fc8e999d6449-multus-cni-dir\") pod \"multus-t4bpj\" (UID: \"140ff0c3-223c-4f7a-9142-fc8e999d6449\") " pod="openshift-multus/multus-t4bpj" Sep 07 16:19:33.458260 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.457292 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcgl7\" (UniqueName: \"kubernetes.io/projected/552ec3ab-d017-450f-85bb-616f65146d2e-kube-api-access-kcgl7\") pod \"network-check-target-8g844\" (UID: \"552ec3ab-d017-450f-85bb-616f65146d2e\") " pod="openshift-network-diagnostics/network-check-target-8g844" Sep 07 16:19:33.459055 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.457293 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/140ff0c3-223c-4f7a-9142-fc8e999d6449-multus-conf-dir\") pod \"multus-t4bpj\" (UID: \"140ff0c3-223c-4f7a-9142-fc8e999d6449\") " pod="openshift-multus/multus-t4bpj" Sep 07 16:19:33.459055 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.457341 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c1912e17-6316-4af6-9eda-b597816af7e8-registration-dir\") pod \"aws-ebs-csi-driver-node-j6dwj\" (UID: \"c1912e17-6316-4af6-9eda-b597816af7e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j6dwj" Sep 07 16:19:33.459055 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.457368 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4f4aaec6-7ab4-48ab-a0e1-4d26977f0315-dbus\") pod \"global-pull-secret-syncer-6n92f\" (UID: \"4f4aaec6-7ab4-48ab-a0e1-4d26977f0315\") " pod="kube-system/global-pull-secret-syncer-6n92f" Sep 07 16:19:33.459055 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.457394 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4f4aaec6-7ab4-48ab-a0e1-4d26977f0315-original-pull-secret\") pod \"global-pull-secret-syncer-6n92f\" (UID: \"4f4aaec6-7ab4-48ab-a0e1-4d26977f0315\") " pod="kube-system/global-pull-secret-syncer-6n92f" Sep 07 16:19:33.459055 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.457426 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2e77c59f-ef45-4fb8-99be-5fdccb594e0c-cni-binary-copy\") pod \"multus-additional-cni-plugins-7tdcz\" (UID: \"2e77c59f-ef45-4fb8-99be-5fdccb594e0c\") " pod="openshift-multus/multus-additional-cni-plugins-7tdcz" Sep 07 16:19:33.459055 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.457450 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2e77c59f-ef45-4fb8-99be-5fdccb594e0c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7tdcz\" (UID: \"2e77c59f-ef45-4fb8-99be-5fdccb594e0c\") " pod="openshift-multus/multus-additional-cni-plugins-7tdcz" Sep 07 16:19:33.459055 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.457467 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c1912e17-6316-4af6-9eda-b597816af7e8-registration-dir\") pod \"aws-ebs-csi-driver-node-j6dwj\" (UID: \"c1912e17-6316-4af6-9eda-b597816af7e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j6dwj" Sep 07 16:19:33.459055 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.457477 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/140ff0c3-223c-4f7a-9142-fc8e999d6449-cnibin\") pod \"multus-t4bpj\" (UID: \"140ff0c3-223c-4f7a-9142-fc8e999d6449\") " pod="openshift-multus/multus-t4bpj" Sep 07 16:19:33.459055 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.457493 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9039f150-f93a-482e-aa48-46844ac5c7bc-host-kubelet\") pod \"ovnkube-node-rz4mc\" (UID: \"9039f150-f93a-482e-aa48-46844ac5c7bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:33.459055 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.457536 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/140ff0c3-223c-4f7a-9142-fc8e999d6449-cnibin\") pod \"multus-t4bpj\" (UID: \"140ff0c3-223c-4f7a-9142-fc8e999d6449\") " pod="openshift-multus/multus-t4bpj" Sep 07 16:19:33.459055 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.457542 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/140ff0c3-223c-4f7a-9142-fc8e999d6449-host-run-netns\") pod \"multus-t4bpj\" (UID: \"140ff0c3-223c-4f7a-9142-fc8e999d6449\") " pod="openshift-multus/multus-t4bpj" Sep 07 16:19:33.459055 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.457575 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/140ff0c3-223c-4f7a-9142-fc8e999d6449-hostroot\") pod \"multus-t4bpj\" (UID: \"140ff0c3-223c-4f7a-9142-fc8e999d6449\") " pod="openshift-multus/multus-t4bpj" Sep 07 16:19:33.459055 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.457584 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/140ff0c3-223c-4f7a-9142-fc8e999d6449-host-run-netns\") pod \"multus-t4bpj\" (UID: \"140ff0c3-223c-4f7a-9142-fc8e999d6449\") " pod="openshift-multus/multus-t4bpj" Sep 07 16:19:33.459055 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.457643 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4013a04f-2b59-44fa-af92-aece71a27fc8-agent-certs\") pod \"konnectivity-agent-hv5bm\" (UID: \"4013a04f-2b59-44fa-af92-aece71a27fc8\") " pod="kube-system/konnectivity-agent-hv5bm" Sep 07 16:19:33.459055 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.457687 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4f4aaec6-7ab4-48ab-a0e1-4d26977f0315-kubelet-config\") pod \"global-pull-secret-syncer-6n92f\" (UID: \"4f4aaec6-7ab4-48ab-a0e1-4d26977f0315\") " pod="kube-system/global-pull-secret-syncer-6n92f" Sep 07 16:19:33.459055 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.457698 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/140ff0c3-223c-4f7a-9142-fc8e999d6449-hostroot\") pod \"multus-t4bpj\" (UID: \"140ff0c3-223c-4f7a-9142-fc8e999d6449\") " pod="openshift-multus/multus-t4bpj" Sep 07 16:19:33.459055 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.457722 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9039f150-f93a-482e-aa48-46844ac5c7bc-host-slash\") pod \"ovnkube-node-rz4mc\" (UID: \"9039f150-f93a-482e-aa48-46844ac5c7bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:33.459809 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.457754 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9039f150-f93a-482e-aa48-46844ac5c7bc-host-run-netns\") pod \"ovnkube-node-rz4mc\" (UID: \"9039f150-f93a-482e-aa48-46844ac5c7bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:33.459809 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.457763 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9039f150-f93a-482e-aa48-46844ac5c7bc-host-slash\") pod \"ovnkube-node-rz4mc\" (UID: \"9039f150-f93a-482e-aa48-46844ac5c7bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:33.459809 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.457801 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9039f150-f93a-482e-aa48-46844ac5c7bc-log-socket\") pod \"ovnkube-node-rz4mc\" (UID: \"9039f150-f93a-482e-aa48-46844ac5c7bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:33.459809 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.457840 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9039f150-f93a-482e-aa48-46844ac5c7bc-log-socket\") pod \"ovnkube-node-rz4mc\" (UID: \"9039f150-f93a-482e-aa48-46844ac5c7bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:33.459809 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.457849 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9039f150-f93a-482e-aa48-46844ac5c7bc-host-run-netns\") pod \"ovnkube-node-rz4mc\" (UID: \"9039f150-f93a-482e-aa48-46844ac5c7bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:33.459809 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.457867 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/553ed3c9-3691-4532-8a44-702ea20b17a7-tmp-dir\") pod \"node-resolver-kcb7q\" (UID: \"553ed3c9-3691-4532-8a44-702ea20b17a7\") " pod="openshift-dns/node-resolver-kcb7q" Sep 07 16:19:33.459809 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.457868 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9039f150-f93a-482e-aa48-46844ac5c7bc-ovnkube-config\") pod \"ovnkube-node-rz4mc\" (UID: \"9039f150-f93a-482e-aa48-46844ac5c7bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:33.459809 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.457902 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/53ae957e-ec96-4d71-bca5-908c965ec61a-tmp\") pod \"tuned-fwmt5\" (UID: \"53ae957e-ec96-4d71-bca5-908c965ec61a\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmt5" Sep 07 16:19:33.459809 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.457927 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwbjt\" (UniqueName: \"kubernetes.io/projected/53ae957e-ec96-4d71-bca5-908c965ec61a-kube-api-access-vwbjt\") pod \"tuned-fwmt5\" (UID: \"53ae957e-ec96-4d71-bca5-908c965ec61a\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmt5" Sep 07 16:19:33.460636 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.460612 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9039f150-f93a-482e-aa48-46844ac5c7bc-ovn-node-metrics-cert\") pod \"ovnkube-node-rz4mc\" (UID: \"9039f150-f93a-482e-aa48-46844ac5c7bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:33.462728 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.462705 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckx28\" (UniqueName: \"kubernetes.io/projected/c1912e17-6316-4af6-9eda-b597816af7e8-kube-api-access-ckx28\") pod \"aws-ebs-csi-driver-node-j6dwj\" (UID: \"c1912e17-6316-4af6-9eda-b597816af7e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j6dwj" Sep 07 16:19:33.463926 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.463903 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxt2d\" (UniqueName: \"kubernetes.io/projected/140ff0c3-223c-4f7a-9142-fc8e999d6449-kube-api-access-fxt2d\") pod \"multus-t4bpj\" (UID: \"140ff0c3-223c-4f7a-9142-fc8e999d6449\") " pod="openshift-multus/multus-t4bpj" Sep 07 16:19:33.465038 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:33.465001 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 07 16:19:33.465144 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:33.465047 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 07 16:19:33.465144 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:33.465058 2580 projected.go:196] Error preparing data for projected volume kube-api-access-kcgl7 for pod openshift-network-diagnostics/network-check-target-8g844: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 16:19:33.465144 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:33.465121 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/552ec3ab-d017-450f-85bb-616f65146d2e-kube-api-access-kcgl7 podName:552ec3ab-d017-450f-85bb-616f65146d2e nodeName:}" failed. No retries permitted until 2026-09-07 16:19:33.965102353 +0000 UTC m=+3.000325417 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-kcgl7" (UniqueName: "kubernetes.io/projected/552ec3ab-d017-450f-85bb-616f65146d2e-kube-api-access-kcgl7") pod "network-check-target-8g844" (UID: "552ec3ab-d017-450f-85bb-616f65146d2e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 16:19:33.468344 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.468313 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltwtv\" (UniqueName: \"kubernetes.io/projected/9039f150-f93a-482e-aa48-46844ac5c7bc-kube-api-access-ltwtv\") pod \"ovnkube-node-rz4mc\" (UID: \"9039f150-f93a-482e-aa48-46844ac5c7bc\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:33.558374 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.558333 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/53ae957e-ec96-4d71-bca5-908c965ec61a-tmp\") pod \"tuned-fwmt5\" (UID: \"53ae957e-ec96-4d71-bca5-908c965ec61a\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmt5" Sep 07 16:19:33.558552 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.558381 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwbjt\" (UniqueName: \"kubernetes.io/projected/53ae957e-ec96-4d71-bca5-908c965ec61a-kube-api-access-vwbjt\") pod \"tuned-fwmt5\" (UID: \"53ae957e-ec96-4d71-bca5-908c965ec61a\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmt5" Sep 07 16:19:33.558552 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.558412 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/53ae957e-ec96-4d71-bca5-908c965ec61a-etc-sysctl-conf\") pod \"tuned-fwmt5\" (UID: \"53ae957e-ec96-4d71-bca5-908c965ec61a\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmt5" Sep 07 16:19:33.558552 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.558435 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/53ae957e-ec96-4d71-bca5-908c965ec61a-host\") pod \"tuned-fwmt5\" (UID: \"53ae957e-ec96-4d71-bca5-908c965ec61a\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmt5" Sep 07 16:19:33.558552 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.558459 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2e77c59f-ef45-4fb8-99be-5fdccb594e0c-cnibin\") pod \"multus-additional-cni-plugins-7tdcz\" (UID: \"2e77c59f-ef45-4fb8-99be-5fdccb594e0c\") " pod="openshift-multus/multus-additional-cni-plugins-7tdcz" Sep 07 16:19:33.558552 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.558482 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/53ae957e-ec96-4d71-bca5-908c965ec61a-etc-tuned\") pod \"tuned-fwmt5\" (UID: \"53ae957e-ec96-4d71-bca5-908c965ec61a\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmt5" Sep 07 16:19:33.558552 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.558504 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8c534a46-aee8-4f33-a447-c049fb3d3f04-serviceca\") pod \"node-ca-g9wgk\" (UID: \"8c534a46-aee8-4f33-a447-c049fb3d3f04\") " pod="openshift-image-registry/node-ca-g9wgk" Sep 07 16:19:33.558552 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.558527 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hm2q\" (UniqueName: \"kubernetes.io/projected/2e77c59f-ef45-4fb8-99be-5fdccb594e0c-kube-api-access-2hm2q\") pod \"multus-additional-cni-plugins-7tdcz\" (UID: \"2e77c59f-ef45-4fb8-99be-5fdccb594e0c\") " pod="openshift-multus/multus-additional-cni-plugins-7tdcz" Sep 07 16:19:33.558835 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.558558 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9dd316ae-6ec2-4359-887d-7e2665504ce4-host-slash\") pod \"iptables-alerter-l9vhv\" (UID: \"9dd316ae-6ec2-4359-887d-7e2665504ce4\") " pod="openshift-network-operator/iptables-alerter-l9vhv" Sep 07 16:19:33.558835 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.558585 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/553ed3c9-3691-4532-8a44-702ea20b17a7-hosts-file\") pod \"node-resolver-kcb7q\" (UID: \"553ed3c9-3691-4532-8a44-702ea20b17a7\") " pod="openshift-dns/node-resolver-kcb7q" Sep 07 16:19:33.558835 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.558608 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/53ae957e-ec96-4d71-bca5-908c965ec61a-etc-sysctl-d\") pod \"tuned-fwmt5\" (UID: \"53ae957e-ec96-4d71-bca5-908c965ec61a\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmt5" Sep 07 16:19:33.558835 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.558602 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/53ae957e-ec96-4d71-bca5-908c965ec61a-etc-sysctl-conf\") pod \"tuned-fwmt5\" (UID: \"53ae957e-ec96-4d71-bca5-908c965ec61a\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmt5" Sep 07 16:19:33.558835 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.558687 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/53ae957e-ec96-4d71-bca5-908c965ec61a-host\") pod \"tuned-fwmt5\" (UID: \"53ae957e-ec96-4d71-bca5-908c965ec61a\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmt5" Sep 07 16:19:33.558835 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.558731 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2e77c59f-ef45-4fb8-99be-5fdccb594e0c-cnibin\") pod \"multus-additional-cni-plugins-7tdcz\" (UID: \"2e77c59f-ef45-4fb8-99be-5fdccb594e0c\") " pod="openshift-multus/multus-additional-cni-plugins-7tdcz" Sep 07 16:19:33.558835 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.558776 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9dd316ae-6ec2-4359-887d-7e2665504ce4-host-slash\") pod \"iptables-alerter-l9vhv\" (UID: \"9dd316ae-6ec2-4359-887d-7e2665504ce4\") " pod="openshift-network-operator/iptables-alerter-l9vhv" Sep 07 16:19:33.560690 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.558912 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/53ae957e-ec96-4d71-bca5-908c965ec61a-etc-systemd\") pod \"tuned-fwmt5\" (UID: \"53ae957e-ec96-4d71-bca5-908c965ec61a\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmt5" Sep 07 16:19:33.560690 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.558948 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn5wf\" (UniqueName: \"kubernetes.io/projected/8c534a46-aee8-4f33-a447-c049fb3d3f04-kube-api-access-sn5wf\") pod \"node-ca-g9wgk\" (UID: \"8c534a46-aee8-4f33-a447-c049fb3d3f04\") " pod="openshift-image-registry/node-ca-g9wgk" Sep 07 16:19:33.560690 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.558978 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjzmg\" (UniqueName: \"kubernetes.io/projected/9dd316ae-6ec2-4359-887d-7e2665504ce4-kube-api-access-xjzmg\") pod \"iptables-alerter-l9vhv\" (UID: \"9dd316ae-6ec2-4359-887d-7e2665504ce4\") " pod="openshift-network-operator/iptables-alerter-l9vhv" Sep 07 16:19:33.560690 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.559002 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4013a04f-2b59-44fa-af92-aece71a27fc8-konnectivity-ca\") pod \"konnectivity-agent-hv5bm\" (UID: \"4013a04f-2b59-44fa-af92-aece71a27fc8\") " pod="kube-system/konnectivity-agent-hv5bm" Sep 07 16:19:33.560690 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.559046 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/53ae957e-ec96-4d71-bca5-908c965ec61a-lib-modules\") pod \"tuned-fwmt5\" (UID: \"53ae957e-ec96-4d71-bca5-908c965ec61a\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmt5" Sep 07 16:19:33.560690 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.559072 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2e77c59f-ef45-4fb8-99be-5fdccb594e0c-system-cni-dir\") pod \"multus-additional-cni-plugins-7tdcz\" (UID: \"2e77c59f-ef45-4fb8-99be-5fdccb594e0c\") " pod="openshift-multus/multus-additional-cni-plugins-7tdcz" Sep 07 16:19:33.560690 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.559098 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2e77c59f-ef45-4fb8-99be-5fdccb594e0c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7tdcz\" (UID: \"2e77c59f-ef45-4fb8-99be-5fdccb594e0c\") " pod="openshift-multus/multus-additional-cni-plugins-7tdcz" Sep 07 16:19:33.560690 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.559132 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/53ae957e-ec96-4d71-bca5-908c965ec61a-etc-modprobe-d\") pod \"tuned-fwmt5\" (UID: \"53ae957e-ec96-4d71-bca5-908c965ec61a\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmt5" Sep 07 16:19:33.560690 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.559163 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/53ae957e-ec96-4d71-bca5-908c965ec61a-etc-kubernetes\") pod \"tuned-fwmt5\" (UID: \"53ae957e-ec96-4d71-bca5-908c965ec61a\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmt5" Sep 07 16:19:33.560690 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.559179 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/53ae957e-ec96-4d71-bca5-908c965ec61a-etc-sysctl-d\") pod \"tuned-fwmt5\" (UID: \"53ae957e-ec96-4d71-bca5-908c965ec61a\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmt5" Sep 07 16:19:33.560690 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.559189 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2e77c59f-ef45-4fb8-99be-5fdccb594e0c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7tdcz\" (UID: \"2e77c59f-ef45-4fb8-99be-5fdccb594e0c\") " pod="openshift-multus/multus-additional-cni-plugins-7tdcz" Sep 07 16:19:33.560690 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.559230 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8c534a46-aee8-4f33-a447-c049fb3d3f04-host\") pod \"node-ca-g9wgk\" (UID: \"8c534a46-aee8-4f33-a447-c049fb3d3f04\") " pod="openshift-image-registry/node-ca-g9wgk" Sep 07 16:19:33.560690 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.559253 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/599e026d-7338-4319-a1d7-7b714bbb6e58-metrics-certs\") pod \"network-metrics-daemon-lhv86\" (UID: \"599e026d-7338-4319-a1d7-7b714bbb6e58\") " pod="openshift-multus/network-metrics-daemon-lhv86" Sep 07 16:19:33.560690 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.559282 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwskj\" (UniqueName: \"kubernetes.io/projected/599e026d-7338-4319-a1d7-7b714bbb6e58-kube-api-access-qwskj\") pod \"network-metrics-daemon-lhv86\" (UID: \"599e026d-7338-4319-a1d7-7b714bbb6e58\") " pod="openshift-multus/network-metrics-daemon-lhv86" Sep 07 16:19:33.560690 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.559309 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9dd316ae-6ec2-4359-887d-7e2665504ce4-iptables-alerter-script\") pod \"iptables-alerter-l9vhv\" (UID: \"9dd316ae-6ec2-4359-887d-7e2665504ce4\") " pod="openshift-network-operator/iptables-alerter-l9vhv" Sep 07 16:19:33.560690 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.559335 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/53ae957e-ec96-4d71-bca5-908c965ec61a-run\") pod \"tuned-fwmt5\" (UID: \"53ae957e-ec96-4d71-bca5-908c965ec61a\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmt5" Sep 07 16:19:33.560690 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.559450 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8c534a46-aee8-4f33-a447-c049fb3d3f04-serviceca\") pod \"node-ca-g9wgk\" (UID: \"8c534a46-aee8-4f33-a447-c049fb3d3f04\") " pod="openshift-image-registry/node-ca-g9wgk" Sep 07 16:19:33.561550 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.559450 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr5zh\" (UniqueName: \"kubernetes.io/projected/553ed3c9-3691-4532-8a44-702ea20b17a7-kube-api-access-zr5zh\") pod \"node-resolver-kcb7q\" (UID: \"553ed3c9-3691-4532-8a44-702ea20b17a7\") " pod="openshift-dns/node-resolver-kcb7q" Sep 07 16:19:33.561550 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.559551 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/53ae957e-ec96-4d71-bca5-908c965ec61a-etc-sysconfig\") pod \"tuned-fwmt5\" (UID: \"53ae957e-ec96-4d71-bca5-908c965ec61a\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmt5" Sep 07 16:19:33.561550 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.559574 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/53ae957e-ec96-4d71-bca5-908c965ec61a-sys\") pod \"tuned-fwmt5\" (UID: \"53ae957e-ec96-4d71-bca5-908c965ec61a\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmt5" Sep 07 16:19:33.561550 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.559596 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/53ae957e-ec96-4d71-bca5-908c965ec61a-var-lib-kubelet\") pod \"tuned-fwmt5\" (UID: \"53ae957e-ec96-4d71-bca5-908c965ec61a\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmt5" Sep 07 16:19:33.561550 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.559598 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/53ae957e-ec96-4d71-bca5-908c965ec61a-etc-modprobe-d\") pod \"tuned-fwmt5\" (UID: \"53ae957e-ec96-4d71-bca5-908c965ec61a\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmt5" Sep 07 16:19:33.561550 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.559628 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4013a04f-2b59-44fa-af92-aece71a27fc8-konnectivity-ca\") pod \"konnectivity-agent-hv5bm\" (UID: \"4013a04f-2b59-44fa-af92-aece71a27fc8\") " pod="kube-system/konnectivity-agent-hv5bm" Sep 07 16:19:33.561550 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.559649 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4f4aaec6-7ab4-48ab-a0e1-4d26977f0315-dbus\") pod \"global-pull-secret-syncer-6n92f\" (UID: \"4f4aaec6-7ab4-48ab-a0e1-4d26977f0315\") " pod="kube-system/global-pull-secret-syncer-6n92f" Sep 07 16:19:33.561550 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.559652 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2e77c59f-ef45-4fb8-99be-5fdccb594e0c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7tdcz\" (UID: \"2e77c59f-ef45-4fb8-99be-5fdccb594e0c\") " pod="openshift-multus/multus-additional-cni-plugins-7tdcz" Sep 07 16:19:33.561550 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.559661 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/53ae957e-ec96-4d71-bca5-908c965ec61a-var-lib-kubelet\") pod \"tuned-fwmt5\" (UID: \"53ae957e-ec96-4d71-bca5-908c965ec61a\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmt5" Sep 07 16:19:33.561550 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.559698 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4f4aaec6-7ab4-48ab-a0e1-4d26977f0315-original-pull-secret\") pod \"global-pull-secret-syncer-6n92f\" (UID: \"4f4aaec6-7ab4-48ab-a0e1-4d26977f0315\") " pod="kube-system/global-pull-secret-syncer-6n92f" Sep 07 16:19:33.561550 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.559716 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4f4aaec6-7ab4-48ab-a0e1-4d26977f0315-dbus\") pod \"global-pull-secret-syncer-6n92f\" (UID: \"4f4aaec6-7ab4-48ab-a0e1-4d26977f0315\") " pod="kube-system/global-pull-secret-syncer-6n92f" Sep 07 16:19:33.561550 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.559728 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2e77c59f-ef45-4fb8-99be-5fdccb594e0c-cni-binary-copy\") pod \"multus-additional-cni-plugins-7tdcz\" (UID: \"2e77c59f-ef45-4fb8-99be-5fdccb594e0c\") " pod="openshift-multus/multus-additional-cni-plugins-7tdcz" Sep 07 16:19:33.561550 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.559735 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/53ae957e-ec96-4d71-bca5-908c965ec61a-etc-systemd\") pod \"tuned-fwmt5\" (UID: \"53ae957e-ec96-4d71-bca5-908c965ec61a\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmt5" Sep 07 16:19:33.561550 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:33.559747 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 16:19:33.561550 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.559757 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2e77c59f-ef45-4fb8-99be-5fdccb594e0c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7tdcz\" (UID: \"2e77c59f-ef45-4fb8-99be-5fdccb594e0c\") " pod="openshift-multus/multus-additional-cni-plugins-7tdcz" Sep 07 16:19:33.561550 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.559762 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/53ae957e-ec96-4d71-bca5-908c965ec61a-etc-kubernetes\") pod \"tuned-fwmt5\" (UID: \"53ae957e-ec96-4d71-bca5-908c965ec61a\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmt5" Sep 07 16:19:33.561550 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.559072 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/553ed3c9-3691-4532-8a44-702ea20b17a7-hosts-file\") pod \"node-resolver-kcb7q\" (UID: \"553ed3c9-3691-4532-8a44-702ea20b17a7\") " pod="openshift-dns/node-resolver-kcb7q" Sep 07 16:19:33.561550 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:33.559818 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 07 16:19:33.562422 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.559505 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8c534a46-aee8-4f33-a447-c049fb3d3f04-host\") pod \"node-ca-g9wgk\" (UID: \"8c534a46-aee8-4f33-a447-c049fb3d3f04\") " pod="openshift-image-registry/node-ca-g9wgk" Sep 07 16:19:33.562422 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:33.559890 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f4aaec6-7ab4-48ab-a0e1-4d26977f0315-original-pull-secret podName:4f4aaec6-7ab4-48ab-a0e1-4d26977f0315 nodeName:}" failed. No retries permitted until 2026-09-07 16:19:34.059869514 +0000 UTC m=+3.095092592 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4f4aaec6-7ab4-48ab-a0e1-4d26977f0315-original-pull-secret") pod "global-pull-secret-syncer-6n92f" (UID: "4f4aaec6-7ab4-48ab-a0e1-4d26977f0315") : object "kube-system"/"original-pull-secret" not registered Sep 07 16:19:33.562422 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:33.559908 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/599e026d-7338-4319-a1d7-7b714bbb6e58-metrics-certs podName:599e026d-7338-4319-a1d7-7b714bbb6e58 nodeName:}" failed. No retries permitted until 2026-09-07 16:19:34.059898779 +0000 UTC m=+3.095121828 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/599e026d-7338-4319-a1d7-7b714bbb6e58-metrics-certs") pod "network-metrics-daemon-lhv86" (UID: "599e026d-7338-4319-a1d7-7b714bbb6e58") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 16:19:33.562422 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.559936 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/53ae957e-ec96-4d71-bca5-908c965ec61a-etc-sysconfig\") pod \"tuned-fwmt5\" (UID: \"53ae957e-ec96-4d71-bca5-908c965ec61a\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmt5" Sep 07 16:19:33.562422 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.559955 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2e77c59f-ef45-4fb8-99be-5fdccb594e0c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7tdcz\" (UID: \"2e77c59f-ef45-4fb8-99be-5fdccb594e0c\") " pod="openshift-multus/multus-additional-cni-plugins-7tdcz" Sep 07 16:19:33.562422 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.559984 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/53ae957e-ec96-4d71-bca5-908c965ec61a-sys\") pod \"tuned-fwmt5\" (UID: \"53ae957e-ec96-4d71-bca5-908c965ec61a\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmt5" Sep 07 16:19:33.562422 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.559997 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2e77c59f-ef45-4fb8-99be-5fdccb594e0c-system-cni-dir\") pod \"multus-additional-cni-plugins-7tdcz\" (UID: \"2e77c59f-ef45-4fb8-99be-5fdccb594e0c\") " pod="openshift-multus/multus-additional-cni-plugins-7tdcz" Sep 07 16:19:33.562422 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.560111 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/53ae957e-ec96-4d71-bca5-908c965ec61a-lib-modules\") pod \"tuned-fwmt5\" (UID: \"53ae957e-ec96-4d71-bca5-908c965ec61a\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmt5" Sep 07 16:19:33.562422 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.560113 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/53ae957e-ec96-4d71-bca5-908c965ec61a-run\") pod \"tuned-fwmt5\" (UID: \"53ae957e-ec96-4d71-bca5-908c965ec61a\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmt5" Sep 07 16:19:33.562422 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.560112 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2e77c59f-ef45-4fb8-99be-5fdccb594e0c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7tdcz\" (UID: \"2e77c59f-ef45-4fb8-99be-5fdccb594e0c\") " pod="openshift-multus/multus-additional-cni-plugins-7tdcz" Sep 07 16:19:33.562422 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.560136 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4013a04f-2b59-44fa-af92-aece71a27fc8-agent-certs\") pod \"konnectivity-agent-hv5bm\" (UID: \"4013a04f-2b59-44fa-af92-aece71a27fc8\") " pod="kube-system/konnectivity-agent-hv5bm" Sep 07 16:19:33.562422 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.560167 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4f4aaec6-7ab4-48ab-a0e1-4d26977f0315-kubelet-config\") pod \"global-pull-secret-syncer-6n92f\" (UID: \"4f4aaec6-7ab4-48ab-a0e1-4d26977f0315\") " pod="kube-system/global-pull-secret-syncer-6n92f" Sep 07 16:19:33.562422 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.560196 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/553ed3c9-3691-4532-8a44-702ea20b17a7-tmp-dir\") pod \"node-resolver-kcb7q\" (UID: \"553ed3c9-3691-4532-8a44-702ea20b17a7\") " pod="openshift-dns/node-resolver-kcb7q" Sep 07 16:19:33.562422 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.560492 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/553ed3c9-3691-4532-8a44-702ea20b17a7-tmp-dir\") pod \"node-resolver-kcb7q\" (UID: \"553ed3c9-3691-4532-8a44-702ea20b17a7\") " pod="openshift-dns/node-resolver-kcb7q" Sep 07 16:19:33.562422 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.560514 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4f4aaec6-7ab4-48ab-a0e1-4d26977f0315-kubelet-config\") pod \"global-pull-secret-syncer-6n92f\" (UID: \"4f4aaec6-7ab4-48ab-a0e1-4d26977f0315\") " pod="kube-system/global-pull-secret-syncer-6n92f" Sep 07 16:19:33.562422 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.560524 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2e77c59f-ef45-4fb8-99be-5fdccb594e0c-cni-binary-copy\") pod \"multus-additional-cni-plugins-7tdcz\" (UID: \"2e77c59f-ef45-4fb8-99be-5fdccb594e0c\") " pod="openshift-multus/multus-additional-cni-plugins-7tdcz" Sep 07 16:19:33.563212 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.560582 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9dd316ae-6ec2-4359-887d-7e2665504ce4-iptables-alerter-script\") pod \"iptables-alerter-l9vhv\" (UID: \"9dd316ae-6ec2-4359-887d-7e2665504ce4\") " pod="openshift-network-operator/iptables-alerter-l9vhv" Sep 07 16:19:33.563212 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.561066 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/53ae957e-ec96-4d71-bca5-908c965ec61a-tmp\") pod \"tuned-fwmt5\" (UID: \"53ae957e-ec96-4d71-bca5-908c965ec61a\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmt5" Sep 07 16:19:33.563212 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.561475 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/53ae957e-ec96-4d71-bca5-908c965ec61a-etc-tuned\") pod \"tuned-fwmt5\" (UID: \"53ae957e-ec96-4d71-bca5-908c965ec61a\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmt5" Sep 07 16:19:33.563212 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.562650 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4013a04f-2b59-44fa-af92-aece71a27fc8-agent-certs\") pod \"konnectivity-agent-hv5bm\" (UID: \"4013a04f-2b59-44fa-af92-aece71a27fc8\") " pod="kube-system/konnectivity-agent-hv5bm" Sep 07 16:19:33.566979 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.566952 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn5wf\" (UniqueName: \"kubernetes.io/projected/8c534a46-aee8-4f33-a447-c049fb3d3f04-kube-api-access-sn5wf\") pod \"node-ca-g9wgk\" (UID: \"8c534a46-aee8-4f33-a447-c049fb3d3f04\") " pod="openshift-image-registry/node-ca-g9wgk" Sep 07 16:19:33.567192 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.567087 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwbjt\" (UniqueName: \"kubernetes.io/projected/53ae957e-ec96-4d71-bca5-908c965ec61a-kube-api-access-vwbjt\") pod \"tuned-fwmt5\" (UID: \"53ae957e-ec96-4d71-bca5-908c965ec61a\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmt5" Sep 07 16:19:33.567951 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.567928 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hm2q\" (UniqueName: \"kubernetes.io/projected/2e77c59f-ef45-4fb8-99be-5fdccb594e0c-kube-api-access-2hm2q\") pod \"multus-additional-cni-plugins-7tdcz\" (UID: \"2e77c59f-ef45-4fb8-99be-5fdccb594e0c\") " pod="openshift-multus/multus-additional-cni-plugins-7tdcz" Sep 07 16:19:33.569006 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.568986 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr5zh\" (UniqueName: \"kubernetes.io/projected/553ed3c9-3691-4532-8a44-702ea20b17a7-kube-api-access-zr5zh\") pod \"node-resolver-kcb7q\" (UID: \"553ed3c9-3691-4532-8a44-702ea20b17a7\") " pod="openshift-dns/node-resolver-kcb7q" Sep 07 16:19:33.569468 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.569453 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjzmg\" (UniqueName: \"kubernetes.io/projected/9dd316ae-6ec2-4359-887d-7e2665504ce4-kube-api-access-xjzmg\") pod \"iptables-alerter-l9vhv\" (UID: \"9dd316ae-6ec2-4359-887d-7e2665504ce4\") " pod="openshift-network-operator/iptables-alerter-l9vhv" Sep 07 16:19:33.569745 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.569726 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwskj\" (UniqueName: \"kubernetes.io/projected/599e026d-7338-4319-a1d7-7b714bbb6e58-kube-api-access-qwskj\") pod \"network-metrics-daemon-lhv86\" (UID: \"599e026d-7338-4319-a1d7-7b714bbb6e58\") " pod="openshift-multus/network-metrics-daemon-lhv86" Sep 07 16:19:33.648090 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.648055 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:33.655690 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.655664 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j6dwj" Sep 07 16:19:33.665855 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.665832 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-t4bpj" Sep 07 16:19:33.671780 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.671762 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-l9vhv" Sep 07 16:19:33.678536 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.678510 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hv5bm" Sep 07 16:19:33.685412 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.685395 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fwmt5" Sep 07 16:19:33.694160 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.694143 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kcb7q" Sep 07 16:19:33.698131 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.698115 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-g9wgk" Sep 07 16:19:33.700960 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.700945 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 07 16:19:33.703638 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:33.703614 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7tdcz" Sep 07 16:19:34.063624 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:34.063590 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/599e026d-7338-4319-a1d7-7b714bbb6e58-metrics-certs\") pod \"network-metrics-daemon-lhv86\" (UID: \"599e026d-7338-4319-a1d7-7b714bbb6e58\") " pod="openshift-multus/network-metrics-daemon-lhv86" Sep 07 16:19:34.063815 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:34.063644 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcgl7\" (UniqueName: \"kubernetes.io/projected/552ec3ab-d017-450f-85bb-616f65146d2e-kube-api-access-kcgl7\") pod \"network-check-target-8g844\" (UID: \"552ec3ab-d017-450f-85bb-616f65146d2e\") " pod="openshift-network-diagnostics/network-check-target-8g844" Sep 07 16:19:34.063815 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:34.063763 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 16:19:34.063815 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:34.063776 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4f4aaec6-7ab4-48ab-a0e1-4d26977f0315-original-pull-secret\") pod \"global-pull-secret-syncer-6n92f\" (UID: \"4f4aaec6-7ab4-48ab-a0e1-4d26977f0315\") " pod="kube-system/global-pull-secret-syncer-6n92f" Sep 07 16:19:34.063962 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:34.063767 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 07 16:19:34.063962 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:34.063837 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/599e026d-7338-4319-a1d7-7b714bbb6e58-metrics-certs podName:599e026d-7338-4319-a1d7-7b714bbb6e58 nodeName:}" failed. No retries permitted until 2026-09-07 16:19:35.063814787 +0000 UTC m=+4.099037855 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/599e026d-7338-4319-a1d7-7b714bbb6e58-metrics-certs") pod "network-metrics-daemon-lhv86" (UID: "599e026d-7338-4319-a1d7-7b714bbb6e58") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 16:19:34.063962 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:34.063859 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 07 16:19:34.063962 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:34.063877 2580 projected.go:196] Error preparing data for projected volume kube-api-access-kcgl7 for pod openshift-network-diagnostics/network-check-target-8g844: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 16:19:34.063962 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:34.063879 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 07 16:19:34.063962 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:34.063936 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f4aaec6-7ab4-48ab-a0e1-4d26977f0315-original-pull-secret podName:4f4aaec6-7ab4-48ab-a0e1-4d26977f0315 nodeName:}" failed. No retries permitted until 2026-09-07 16:19:35.063920494 +0000 UTC m=+4.099143559 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4f4aaec6-7ab4-48ab-a0e1-4d26977f0315-original-pull-secret") pod "global-pull-secret-syncer-6n92f" (UID: "4f4aaec6-7ab4-48ab-a0e1-4d26977f0315") : object "kube-system"/"original-pull-secret" not registered Sep 07 16:19:34.063962 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:34.063957 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/552ec3ab-d017-450f-85bb-616f65146d2e-kube-api-access-kcgl7 podName:552ec3ab-d017-450f-85bb-616f65146d2e nodeName:}" failed. No retries permitted until 2026-09-07 16:19:35.063947771 +0000 UTC m=+4.099170824 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-kcgl7" (UniqueName: "kubernetes.io/projected/552ec3ab-d017-450f-85bb-616f65146d2e-kube-api-access-kcgl7") pod "network-check-target-8g844" (UID: "552ec3ab-d017-450f-85bb-616f65146d2e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 16:19:34.275807 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:34.275774 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c534a46_aee8_4f33_a447_c049fb3d3f04.slice/crio-7beb4fc3391e6d1d5213bd04ad1ab5f512cb84ddacb53b32e857a7afed4167f9 WatchSource:0}: Error finding container 7beb4fc3391e6d1d5213bd04ad1ab5f512cb84ddacb53b32e857a7afed4167f9: Status 404 returned error can't find the container with id 7beb4fc3391e6d1d5213bd04ad1ab5f512cb84ddacb53b32e857a7afed4167f9 Sep 07 16:19:34.282682 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:34.282653 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9039f150_f93a_482e_aa48_46844ac5c7bc.slice/crio-a039b5e235532d50a3f2eee39c0d540bbd9e886740e443cbc9a5540651a2b280 WatchSource:0}: Error finding container a039b5e235532d50a3f2eee39c0d540bbd9e886740e443cbc9a5540651a2b280: Status 404 returned error can't find the container with id a039b5e235532d50a3f2eee39c0d540bbd9e886740e443cbc9a5540651a2b280 Sep 07 16:19:34.283864 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:34.283839 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod140ff0c3_223c_4f7a_9142_fc8e999d6449.slice/crio-10f71b80c43130f039ff170cc056a9a1ea359dc0adde31f494027e9744607b96 WatchSource:0}: Error finding container 10f71b80c43130f039ff170cc056a9a1ea359dc0adde31f494027e9744607b96: Status 404 returned error can't find the container with id 10f71b80c43130f039ff170cc056a9a1ea359dc0adde31f494027e9744607b96 Sep 07 16:19:34.284713 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:34.284692 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e77c59f_ef45_4fb8_99be_5fdccb594e0c.slice/crio-d61508e6a63042e94e9c9b2d29106f8d8c9b57026ea0443f5ff069a0cd20ad62 WatchSource:0}: Error finding container d61508e6a63042e94e9c9b2d29106f8d8c9b57026ea0443f5ff069a0cd20ad62: Status 404 returned error can't find the container with id d61508e6a63042e94e9c9b2d29106f8d8c9b57026ea0443f5ff069a0cd20ad62 Sep 07 16:19:34.285739 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:34.285716 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod553ed3c9_3691_4532_8a44_702ea20b17a7.slice/crio-f36906e0d074b437fa9f43cd27c2e45a186cdb30b6c35e523cf4e94f68126dfd WatchSource:0}: Error finding container f36906e0d074b437fa9f43cd27c2e45a186cdb30b6c35e523cf4e94f68126dfd: Status 404 returned error can't find the container with id f36906e0d074b437fa9f43cd27c2e45a186cdb30b6c35e523cf4e94f68126dfd Sep 07 16:19:34.286927 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:34.286902 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dd316ae_6ec2_4359_887d_7e2665504ce4.slice/crio-b4caee26014f7af61dde4f21c547c0de82199113f1dd1926bab3fa1e17ff11dc WatchSource:0}: Error finding container b4caee26014f7af61dde4f21c547c0de82199113f1dd1926bab3fa1e17ff11dc: Status 404 returned error can't find the container with id b4caee26014f7af61dde4f21c547c0de82199113f1dd1926bab3fa1e17ff11dc Sep 07 16:19:34.288156 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:34.287905 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53ae957e_ec96_4d71_bca5_908c965ec61a.slice/crio-4e087a10d9d3640b031b470119932c09a442b6410c279c2e5adb2d94815f6adf WatchSource:0}: Error finding container 4e087a10d9d3640b031b470119932c09a442b6410c279c2e5adb2d94815f6adf: Status 404 returned error can't find the container with id 4e087a10d9d3640b031b470119932c09a442b6410c279c2e5adb2d94815f6adf Sep 07 16:19:34.289251 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:34.289113 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1912e17_6316_4af6_9eda_b597816af7e8.slice/crio-a00c4051a2ca789895e275772fa6fb5339adb948106747a22201205d7903fd0d WatchSource:0}: Error finding container a00c4051a2ca789895e275772fa6fb5339adb948106747a22201205d7903fd0d: Status 404 returned error can't find the container with id a00c4051a2ca789895e275772fa6fb5339adb948106747a22201205d7903fd0d Sep 07 16:19:34.291199 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:19:34.290698 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4013a04f_2b59_44fa_af92_aece71a27fc8.slice/crio-0f083ef56880e85891eb509969d3b5ad3f1bae78e3dc0fbf1d75de2f359f1f51 WatchSource:0}: Error finding container 0f083ef56880e85891eb509969d3b5ad3f1bae78e3dc0fbf1d75de2f359f1f51: Status 404 returned error can't find the container with id 0f083ef56880e85891eb509969d3b5ad3f1bae78e3dc0fbf1d75de2f359f1f51 Sep 07 16:19:34.385253 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:34.385224 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-06 16:14:33 +0000 UTC" deadline="2028-04-16 19:01:37.516092053 +0000 UTC" Sep 07 16:19:34.385253 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:34.385251 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14090h42m3.130844668s" Sep 07 16:19:34.423182 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:34.423149 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" event={"ID":"9039f150-f93a-482e-aa48-46844ac5c7bc","Type":"ContainerStarted","Data":"a039b5e235532d50a3f2eee39c0d540bbd9e886740e443cbc9a5540651a2b280"} Sep 07 16:19:34.424312 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:34.424284 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hv5bm" event={"ID":"4013a04f-2b59-44fa-af92-aece71a27fc8","Type":"ContainerStarted","Data":"0f083ef56880e85891eb509969d3b5ad3f1bae78e3dc0fbf1d75de2f359f1f51"} Sep 07 16:19:34.425583 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:34.425561 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fwmt5" event={"ID":"53ae957e-ec96-4d71-bca5-908c965ec61a","Type":"ContainerStarted","Data":"4e087a10d9d3640b031b470119932c09a442b6410c279c2e5adb2d94815f6adf"} Sep 07 16:19:34.426647 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:34.426628 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-g9wgk" event={"ID":"8c534a46-aee8-4f33-a447-c049fb3d3f04","Type":"ContainerStarted","Data":"7beb4fc3391e6d1d5213bd04ad1ab5f512cb84ddacb53b32e857a7afed4167f9"} Sep 07 16:19:34.431165 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:34.429226 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-242.ec2.internal" event={"ID":"c620202158ca74d37111884761e47e70","Type":"ContainerStarted","Data":"e4943cea69999fb20b1502ef0702bb6f8fcd1cfd9ae710fcbb0519b0878465e0"} Sep 07 16:19:34.433322 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:34.433298 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j6dwj" event={"ID":"c1912e17-6316-4af6-9eda-b597816af7e8","Type":"ContainerStarted","Data":"a00c4051a2ca789895e275772fa6fb5339adb948106747a22201205d7903fd0d"} Sep 07 16:19:34.435366 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:34.435333 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7tdcz" event={"ID":"2e77c59f-ef45-4fb8-99be-5fdccb594e0c","Type":"ContainerStarted","Data":"d61508e6a63042e94e9c9b2d29106f8d8c9b57026ea0443f5ff069a0cd20ad62"} Sep 07 16:19:34.437296 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:34.437276 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kcb7q" event={"ID":"553ed3c9-3691-4532-8a44-702ea20b17a7","Type":"ContainerStarted","Data":"f36906e0d074b437fa9f43cd27c2e45a186cdb30b6c35e523cf4e94f68126dfd"} Sep 07 16:19:34.439328 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:34.439302 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-l9vhv" event={"ID":"9dd316ae-6ec2-4359-887d-7e2665504ce4","Type":"ContainerStarted","Data":"b4caee26014f7af61dde4f21c547c0de82199113f1dd1926bab3fa1e17ff11dc"} Sep 07 16:19:34.441151 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:34.441129 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-t4bpj" event={"ID":"140ff0c3-223c-4f7a-9142-fc8e999d6449","Type":"ContainerStarted","Data":"10f71b80c43130f039ff170cc056a9a1ea359dc0adde31f494027e9744607b96"} Sep 07 16:19:34.442034 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:34.441974 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-242.ec2.internal" podStartSLOduration=2.441957438 podStartE2EDuration="2.441957438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 16:19:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:19:34.441276936 +0000 UTC m=+3.476500009" watchObservedRunningTime="2026-09-07 16:19:34.441957438 +0000 UTC m=+3.477180512" Sep 07 16:19:35.072777 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:35.072735 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/599e026d-7338-4319-a1d7-7b714bbb6e58-metrics-certs\") pod \"network-metrics-daemon-lhv86\" (UID: \"599e026d-7338-4319-a1d7-7b714bbb6e58\") " pod="openshift-multus/network-metrics-daemon-lhv86" Sep 07 16:19:35.072971 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:35.072805 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcgl7\" (UniqueName: \"kubernetes.io/projected/552ec3ab-d017-450f-85bb-616f65146d2e-kube-api-access-kcgl7\") pod \"network-check-target-8g844\" (UID: \"552ec3ab-d017-450f-85bb-616f65146d2e\") " pod="openshift-network-diagnostics/network-check-target-8g844" Sep 07 16:19:35.072971 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:35.072837 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4f4aaec6-7ab4-48ab-a0e1-4d26977f0315-original-pull-secret\") pod \"global-pull-secret-syncer-6n92f\" (UID: \"4f4aaec6-7ab4-48ab-a0e1-4d26977f0315\") " pod="kube-system/global-pull-secret-syncer-6n92f" Sep 07 16:19:35.073118 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:35.072996 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 07 16:19:35.073118 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:35.073076 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f4aaec6-7ab4-48ab-a0e1-4d26977f0315-original-pull-secret podName:4f4aaec6-7ab4-48ab-a0e1-4d26977f0315 nodeName:}" failed. No retries permitted until 2026-09-07 16:19:37.073057542 +0000 UTC m=+6.108280607 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4f4aaec6-7ab4-48ab-a0e1-4d26977f0315-original-pull-secret") pod "global-pull-secret-syncer-6n92f" (UID: "4f4aaec6-7ab4-48ab-a0e1-4d26977f0315") : object "kube-system"/"original-pull-secret" not registered Sep 07 16:19:35.073507 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:35.073459 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 16:19:35.073507 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:35.073510 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/599e026d-7338-4319-a1d7-7b714bbb6e58-metrics-certs podName:599e026d-7338-4319-a1d7-7b714bbb6e58 nodeName:}" failed. No retries permitted until 2026-09-07 16:19:37.073494646 +0000 UTC m=+6.108717710 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/599e026d-7338-4319-a1d7-7b714bbb6e58-metrics-certs") pod "network-metrics-daemon-lhv86" (UID: "599e026d-7338-4319-a1d7-7b714bbb6e58") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 16:19:35.073658 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:35.073588 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 07 16:19:35.073658 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:35.073605 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 07 16:19:35.073658 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:35.073617 2580 projected.go:196] Error preparing data for projected volume kube-api-access-kcgl7 for pod openshift-network-diagnostics/network-check-target-8g844: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 16:19:35.073658 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:35.073652 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/552ec3ab-d017-450f-85bb-616f65146d2e-kube-api-access-kcgl7 podName:552ec3ab-d017-450f-85bb-616f65146d2e nodeName:}" failed. No retries permitted until 2026-09-07 16:19:37.073640835 +0000 UTC m=+6.108863890 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-kcgl7" (UniqueName: "kubernetes.io/projected/552ec3ab-d017-450f-85bb-616f65146d2e-kube-api-access-kcgl7") pod "network-check-target-8g844" (UID: "552ec3ab-d017-450f-85bb-616f65146d2e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 16:19:35.386180 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:35.386058 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-06 16:14:33 +0000 UTC" deadline="2028-02-03 19:39:45.739365829 +0000 UTC" Sep 07 16:19:35.386180 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:35.386127 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12339h20m10.353243487s" Sep 07 16:19:35.414954 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:35.414920 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhv86" Sep 07 16:19:35.415142 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:35.415084 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhv86" podUID="599e026d-7338-4319-a1d7-7b714bbb6e58" Sep 07 16:19:35.415517 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:35.415490 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6n92f" Sep 07 16:19:35.415613 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:35.415598 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6n92f" podUID="4f4aaec6-7ab4-48ab-a0e1-4d26977f0315" Sep 07 16:19:35.415690 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:35.415664 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8g844" Sep 07 16:19:35.415804 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:35.415742 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8g844" podUID="552ec3ab-d017-450f-85bb-616f65146d2e" Sep 07 16:19:35.482186 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:35.482144 2580 generic.go:356] "Generic (PLEG): container finished" podID="8c1647d1d87b8b8396e8fceb50173ea2" containerID="8c0264ef20dc2d51056b08de63501b25cba22f2511c4b9de450e25d11a4a63c8" exitCode=0 Sep 07 16:19:35.483463 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:35.483103 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-242.ec2.internal" event={"ID":"8c1647d1d87b8b8396e8fceb50173ea2","Type":"ContainerDied","Data":"8c0264ef20dc2d51056b08de63501b25cba22f2511c4b9de450e25d11a4a63c8"} Sep 07 16:19:36.499225 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:36.498772 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-242.ec2.internal" event={"ID":"8c1647d1d87b8b8396e8fceb50173ea2","Type":"ContainerStarted","Data":"9512b229dbd0e5a324f3ee937b7491b9648417b74a10ed0cbe3226da4e7cacba"} Sep 07 16:19:37.090496 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:37.090448 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/599e026d-7338-4319-a1d7-7b714bbb6e58-metrics-certs\") pod \"network-metrics-daemon-lhv86\" (UID: \"599e026d-7338-4319-a1d7-7b714bbb6e58\") " pod="openshift-multus/network-metrics-daemon-lhv86" Sep 07 16:19:37.090691 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:37.090512 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcgl7\" (UniqueName: \"kubernetes.io/projected/552ec3ab-d017-450f-85bb-616f65146d2e-kube-api-access-kcgl7\") pod \"network-check-target-8g844\" (UID: \"552ec3ab-d017-450f-85bb-616f65146d2e\") " pod="openshift-network-diagnostics/network-check-target-8g844" Sep 07 16:19:37.090691 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:37.090540 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4f4aaec6-7ab4-48ab-a0e1-4d26977f0315-original-pull-secret\") pod \"global-pull-secret-syncer-6n92f\" (UID: \"4f4aaec6-7ab4-48ab-a0e1-4d26977f0315\") " pod="kube-system/global-pull-secret-syncer-6n92f" Sep 07 16:19:37.090691 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:37.090576 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 16:19:37.090691 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:37.090650 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/599e026d-7338-4319-a1d7-7b714bbb6e58-metrics-certs podName:599e026d-7338-4319-a1d7-7b714bbb6e58 nodeName:}" failed. No retries permitted until 2026-09-07 16:19:41.090629632 +0000 UTC m=+10.125852685 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/599e026d-7338-4319-a1d7-7b714bbb6e58-metrics-certs") pod "network-metrics-daemon-lhv86" (UID: "599e026d-7338-4319-a1d7-7b714bbb6e58") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 16:19:37.090691 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:37.090650 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 07 16:19:37.090691 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:37.090691 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f4aaec6-7ab4-48ab-a0e1-4d26977f0315-original-pull-secret podName:4f4aaec6-7ab4-48ab-a0e1-4d26977f0315 nodeName:}" failed. No retries permitted until 2026-09-07 16:19:41.090678228 +0000 UTC m=+10.125901277 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4f4aaec6-7ab4-48ab-a0e1-4d26977f0315-original-pull-secret") pod "global-pull-secret-syncer-6n92f" (UID: "4f4aaec6-7ab4-48ab-a0e1-4d26977f0315") : object "kube-system"/"original-pull-secret" not registered Sep 07 16:19:37.090981 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:37.090744 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 07 16:19:37.090981 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:37.090762 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 07 16:19:37.090981 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:37.090775 2580 projected.go:196] Error preparing data for projected volume kube-api-access-kcgl7 for pod openshift-network-diagnostics/network-check-target-8g844: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 16:19:37.090981 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:37.090827 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/552ec3ab-d017-450f-85bb-616f65146d2e-kube-api-access-kcgl7 podName:552ec3ab-d017-450f-85bb-616f65146d2e nodeName:}" failed. No retries permitted until 2026-09-07 16:19:41.090809845 +0000 UTC m=+10.126032908 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-kcgl7" (UniqueName: "kubernetes.io/projected/552ec3ab-d017-450f-85bb-616f65146d2e-kube-api-access-kcgl7") pod "network-check-target-8g844" (UID: "552ec3ab-d017-450f-85bb-616f65146d2e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 16:19:37.413172 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:37.413082 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8g844" Sep 07 16:19:37.413334 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:37.413210 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8g844" podUID="552ec3ab-d017-450f-85bb-616f65146d2e" Sep 07 16:19:37.413603 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:37.413580 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhv86" Sep 07 16:19:37.413717 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:37.413693 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhv86" podUID="599e026d-7338-4319-a1d7-7b714bbb6e58" Sep 07 16:19:37.413870 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:37.413836 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6n92f" Sep 07 16:19:37.413988 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:37.413934 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6n92f" podUID="4f4aaec6-7ab4-48ab-a0e1-4d26977f0315" Sep 07 16:19:39.412919 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:39.412584 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8g844" Sep 07 16:19:39.413389 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:39.413000 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8g844" podUID="552ec3ab-d017-450f-85bb-616f65146d2e" Sep 07 16:19:39.413764 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:39.413483 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhv86" Sep 07 16:19:39.413764 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:39.413586 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhv86" podUID="599e026d-7338-4319-a1d7-7b714bbb6e58" Sep 07 16:19:39.413764 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:39.413654 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6n92f" Sep 07 16:19:39.413764 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:39.413713 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6n92f" podUID="4f4aaec6-7ab4-48ab-a0e1-4d26977f0315" Sep 07 16:19:41.135929 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:41.135038 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcgl7\" (UniqueName: \"kubernetes.io/projected/552ec3ab-d017-450f-85bb-616f65146d2e-kube-api-access-kcgl7\") pod \"network-check-target-8g844\" (UID: \"552ec3ab-d017-450f-85bb-616f65146d2e\") " pod="openshift-network-diagnostics/network-check-target-8g844" Sep 07 16:19:41.135929 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:41.135087 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4f4aaec6-7ab4-48ab-a0e1-4d26977f0315-original-pull-secret\") pod \"global-pull-secret-syncer-6n92f\" (UID: \"4f4aaec6-7ab4-48ab-a0e1-4d26977f0315\") " pod="kube-system/global-pull-secret-syncer-6n92f" Sep 07 16:19:41.135929 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:41.135157 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/599e026d-7338-4319-a1d7-7b714bbb6e58-metrics-certs\") pod \"network-metrics-daemon-lhv86\" (UID: \"599e026d-7338-4319-a1d7-7b714bbb6e58\") " pod="openshift-multus/network-metrics-daemon-lhv86" Sep 07 16:19:41.135929 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:41.135281 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 16:19:41.135929 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:41.135345 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/599e026d-7338-4319-a1d7-7b714bbb6e58-metrics-certs podName:599e026d-7338-4319-a1d7-7b714bbb6e58 nodeName:}" failed. No retries permitted until 2026-09-07 16:19:49.135326885 +0000 UTC m=+18.170549939 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/599e026d-7338-4319-a1d7-7b714bbb6e58-metrics-certs") pod "network-metrics-daemon-lhv86" (UID: "599e026d-7338-4319-a1d7-7b714bbb6e58") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 16:19:41.135929 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:41.135740 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 07 16:19:41.135929 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:41.135753 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 07 16:19:41.135929 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:41.135766 2580 projected.go:196] Error preparing data for projected volume kube-api-access-kcgl7 for pod openshift-network-diagnostics/network-check-target-8g844: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 16:19:41.135929 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:41.135810 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/552ec3ab-d017-450f-85bb-616f65146d2e-kube-api-access-kcgl7 podName:552ec3ab-d017-450f-85bb-616f65146d2e nodeName:}" failed. No retries permitted until 2026-09-07 16:19:49.135795707 +0000 UTC m=+18.171018761 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-kcgl7" (UniqueName: "kubernetes.io/projected/552ec3ab-d017-450f-85bb-616f65146d2e-kube-api-access-kcgl7") pod "network-check-target-8g844" (UID: "552ec3ab-d017-450f-85bb-616f65146d2e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 16:19:41.135929 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:41.135867 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 07 16:19:41.135929 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:41.135892 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f4aaec6-7ab4-48ab-a0e1-4d26977f0315-original-pull-secret podName:4f4aaec6-7ab4-48ab-a0e1-4d26977f0315 nodeName:}" failed. No retries permitted until 2026-09-07 16:19:49.135884685 +0000 UTC m=+18.171107733 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4f4aaec6-7ab4-48ab-a0e1-4d26977f0315-original-pull-secret") pod "global-pull-secret-syncer-6n92f" (UID: "4f4aaec6-7ab4-48ab-a0e1-4d26977f0315") : object "kube-system"/"original-pull-secret" not registered Sep 07 16:19:41.413853 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:41.412991 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8g844" Sep 07 16:19:41.413853 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:41.413064 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6n92f" Sep 07 16:19:41.413853 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:41.413081 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhv86" Sep 07 16:19:41.413853 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:41.413183 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6n92f" podUID="4f4aaec6-7ab4-48ab-a0e1-4d26977f0315" Sep 07 16:19:41.413853 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:41.413632 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhv86" podUID="599e026d-7338-4319-a1d7-7b714bbb6e58" Sep 07 16:19:41.413853 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:41.413764 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8g844" podUID="552ec3ab-d017-450f-85bb-616f65146d2e" Sep 07 16:19:43.412904 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:43.412520 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8g844" Sep 07 16:19:43.412904 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:43.412525 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhv86" Sep 07 16:19:43.412904 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:43.412623 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8g844" podUID="552ec3ab-d017-450f-85bb-616f65146d2e" Sep 07 16:19:43.412904 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:43.412646 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6n92f" Sep 07 16:19:43.412904 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:43.412750 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhv86" podUID="599e026d-7338-4319-a1d7-7b714bbb6e58" Sep 07 16:19:43.412904 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:43.412841 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6n92f" podUID="4f4aaec6-7ab4-48ab-a0e1-4d26977f0315" Sep 07 16:19:45.412772 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:45.412732 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8g844" Sep 07 16:19:45.412772 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:45.412757 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6n92f" Sep 07 16:19:45.413324 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:45.412732 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhv86" Sep 07 16:19:45.413324 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:45.412862 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8g844" podUID="552ec3ab-d017-450f-85bb-616f65146d2e" Sep 07 16:19:45.413324 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:45.412918 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhv86" podUID="599e026d-7338-4319-a1d7-7b714bbb6e58" Sep 07 16:19:45.413324 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:45.412981 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6n92f" podUID="4f4aaec6-7ab4-48ab-a0e1-4d26977f0315" Sep 07 16:19:47.413112 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:47.413068 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhv86" Sep 07 16:19:47.413606 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:47.413119 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8g844" Sep 07 16:19:47.413606 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:47.413179 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6n92f" Sep 07 16:19:47.413606 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:47.413270 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6n92f" podUID="4f4aaec6-7ab4-48ab-a0e1-4d26977f0315" Sep 07 16:19:47.413606 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:47.413314 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8g844" podUID="552ec3ab-d017-450f-85bb-616f65146d2e" Sep 07 16:19:47.413606 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:47.413381 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhv86" podUID="599e026d-7338-4319-a1d7-7b714bbb6e58" Sep 07 16:19:49.203216 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:49.203174 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/599e026d-7338-4319-a1d7-7b714bbb6e58-metrics-certs\") pod \"network-metrics-daemon-lhv86\" (UID: \"599e026d-7338-4319-a1d7-7b714bbb6e58\") " pod="openshift-multus/network-metrics-daemon-lhv86" Sep 07 16:19:49.203216 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:49.203226 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcgl7\" (UniqueName: \"kubernetes.io/projected/552ec3ab-d017-450f-85bb-616f65146d2e-kube-api-access-kcgl7\") pod \"network-check-target-8g844\" (UID: \"552ec3ab-d017-450f-85bb-616f65146d2e\") " pod="openshift-network-diagnostics/network-check-target-8g844" Sep 07 16:19:49.203813 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:49.203336 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 16:19:49.203813 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:49.203349 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4f4aaec6-7ab4-48ab-a0e1-4d26977f0315-original-pull-secret\") pod \"global-pull-secret-syncer-6n92f\" (UID: \"4f4aaec6-7ab4-48ab-a0e1-4d26977f0315\") " pod="kube-system/global-pull-secret-syncer-6n92f" Sep 07 16:19:49.203813 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:49.203340 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 07 16:19:49.203813 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:49.203435 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 07 16:19:49.203813 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:49.203452 2580 projected.go:196] Error preparing data for projected volume kube-api-access-kcgl7 for pod openshift-network-diagnostics/network-check-target-8g844: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 16:19:49.203813 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:49.203460 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 07 16:19:49.203813 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:49.203417 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/599e026d-7338-4319-a1d7-7b714bbb6e58-metrics-certs podName:599e026d-7338-4319-a1d7-7b714bbb6e58 nodeName:}" failed. No retries permitted until 2026-09-07 16:20:05.203396355 +0000 UTC m=+34.238619425 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/599e026d-7338-4319-a1d7-7b714bbb6e58-metrics-certs") pod "network-metrics-daemon-lhv86" (UID: "599e026d-7338-4319-a1d7-7b714bbb6e58") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 16:19:49.203813 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:49.203512 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/552ec3ab-d017-450f-85bb-616f65146d2e-kube-api-access-kcgl7 podName:552ec3ab-d017-450f-85bb-616f65146d2e nodeName:}" failed. No retries permitted until 2026-09-07 16:20:05.20349806 +0000 UTC m=+34.238721112 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-kcgl7" (UniqueName: "kubernetes.io/projected/552ec3ab-d017-450f-85bb-616f65146d2e-kube-api-access-kcgl7") pod "network-check-target-8g844" (UID: "552ec3ab-d017-450f-85bb-616f65146d2e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 16:19:49.203813 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:49.203538 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f4aaec6-7ab4-48ab-a0e1-4d26977f0315-original-pull-secret podName:4f4aaec6-7ab4-48ab-a0e1-4d26977f0315 nodeName:}" failed. No retries permitted until 2026-09-07 16:20:05.20352814 +0000 UTC m=+34.238751191 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4f4aaec6-7ab4-48ab-a0e1-4d26977f0315-original-pull-secret") pod "global-pull-secret-syncer-6n92f" (UID: "4f4aaec6-7ab4-48ab-a0e1-4d26977f0315") : object "kube-system"/"original-pull-secret" not registered Sep 07 16:19:49.412809 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:49.412772 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhv86" Sep 07 16:19:49.412989 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:49.412918 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhv86" podUID="599e026d-7338-4319-a1d7-7b714bbb6e58" Sep 07 16:19:49.412989 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:49.412772 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8g844" Sep 07 16:19:49.413124 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:49.413011 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8g844" podUID="552ec3ab-d017-450f-85bb-616f65146d2e" Sep 07 16:19:49.413124 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:49.412772 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6n92f" Sep 07 16:19:49.413202 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:49.413121 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6n92f" podUID="4f4aaec6-7ab4-48ab-a0e1-4d26977f0315" Sep 07 16:19:51.413412 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:51.413380 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6n92f" Sep 07 16:19:51.413833 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:51.413488 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhv86" Sep 07 16:19:51.413833 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:51.413523 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6n92f" podUID="4f4aaec6-7ab4-48ab-a0e1-4d26977f0315" Sep 07 16:19:51.413833 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:51.413578 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhv86" podUID="599e026d-7338-4319-a1d7-7b714bbb6e58" Sep 07 16:19:51.413833 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:51.413616 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8g844" Sep 07 16:19:51.413833 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:51.413679 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8g844" podUID="552ec3ab-d017-450f-85bb-616f65146d2e" Sep 07 16:19:52.534327 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:52.534299 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-g9wgk" event={"ID":"8c534a46-aee8-4f33-a447-c049fb3d3f04","Type":"ContainerStarted","Data":"b4585adedc98a6362708dd153e1c4658a4478daf16726bbd39cae2cabbd852c2"} Sep 07 16:19:52.546159 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:52.546129 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j6dwj" event={"ID":"c1912e17-6316-4af6-9eda-b597816af7e8","Type":"ContainerStarted","Data":"0fd3f8b64856d128af1b7e4bf59f19a37fbed157d1a2c64b0152ec67f45b9bae"} Sep 07 16:19:52.546276 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:52.546173 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-g9wgk" podStartSLOduration=3.5818189670000002 podStartE2EDuration="21.546161172s" totalImagesPullingTime="17.964342205s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 16:19:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:19:52.546055902 +0000 UTC m=+21.581278967" watchObservedRunningTime="2026-09-07 16:19:52.546161172 +0000 UTC m=+21.581384243" Sep 07 16:19:52.546513 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:52.546491 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-242.ec2.internal" podStartSLOduration=20.546482755 podStartE2EDuration="20.546482755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 16:19:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:19:36.512731941 +0000 UTC m=+5.547955013" watchObservedRunningTime="2026-09-07 16:19:52.546482755 +0000 UTC m=+21.581705826" Sep 07 16:19:52.548112 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:52.548086 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7tdcz" event={"ID":"2e77c59f-ef45-4fb8-99be-5fdccb594e0c","Type":"ContainerStarted","Data":"58cabab85326105658ad1d720be3d27bd670ef7f86fea65f910ba4eb69ad0050"} Sep 07 16:19:52.549406 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:52.549385 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kcb7q" event={"ID":"553ed3c9-3691-4532-8a44-702ea20b17a7","Type":"ContainerStarted","Data":"fe6c3f794936c5ac631c144205f2da24946f74c405b79bd6641a18458b97ff1f"} Sep 07 16:19:52.550702 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:52.550683 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-t4bpj" event={"ID":"140ff0c3-223c-4f7a-9142-fc8e999d6449","Type":"ContainerStarted","Data":"9cba16fbe632a9c1eea7a8676bc48688b6e304f7cbab2ccd3921bd8ae3f222ab"} Sep 07 16:19:52.551913 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:52.551888 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hv5bm" event={"ID":"4013a04f-2b59-44fa-af92-aece71a27fc8","Type":"ContainerStarted","Data":"7ba1cc335b326ff7662485a999b11c8bb083dcbaa1f70bd83b6eb2c8aa38b1a0"} Sep 07 16:19:52.553239 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:52.553219 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fwmt5" event={"ID":"53ae957e-ec96-4d71-bca5-908c965ec61a","Type":"ContainerStarted","Data":"902bd0d4c7833b9fe2737bb256c9e083f418c56f10b4425ddf1a1dbe9694de8c"} Sep 07 16:19:52.582803 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:52.582734 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-hv5bm" podStartSLOduration=7.863114638 podStartE2EDuration="21.582721418s" totalImagesPullingTime="13.71960678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:19:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:19:52.582347343 +0000 UTC m=+21.617570413" watchObservedRunningTime="2026-09-07 16:19:52.582721418 +0000 UTC m=+21.617944489" Sep 07 16:19:52.611635 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:52.611596 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-fwmt5" podStartSLOduration=3.656498648 podStartE2EDuration="21.611584356s" totalImagesPullingTime="17.955085708s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 16:19:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:19:52.596069429 +0000 UTC m=+21.631292500" watchObservedRunningTime="2026-09-07 16:19:52.611584356 +0000 UTC m=+21.646807428" Sep 07 16:19:52.611763 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:52.611743 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-t4bpj" podStartSLOduration=3.619092699 podStartE2EDuration="21.611738881s" totalImagesPullingTime="17.992646182s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 16:19:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:19:52.611163361 +0000 UTC m=+21.646386444" watchObservedRunningTime="2026-09-07 16:19:52.611738881 +0000 UTC m=+21.646961951" Sep 07 16:19:52.628809 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:52.628774 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-kcb7q" podStartSLOduration=7.903703439 podStartE2EDuration="21.628761642s" totalImagesPullingTime="13.725058203s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 16:19:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:19:52.628574801 +0000 UTC m=+21.663797873" watchObservedRunningTime="2026-09-07 16:19:52.628761642 +0000 UTC m=+21.663984712" Sep 07 16:19:52.993960 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:52.993733 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-hv5bm" Sep 07 16:19:52.994855 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:52.994831 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-hv5bm" Sep 07 16:19:53.412447 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:53.412425 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6n92f" Sep 07 16:19:53.412549 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:53.412429 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8g844" Sep 07 16:19:53.412549 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:53.412528 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhv86" Sep 07 16:19:53.412549 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:53.412529 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6n92f" podUID="4f4aaec6-7ab4-48ab-a0e1-4d26977f0315" Sep 07 16:19:53.412738 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:53.412598 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8g844" podUID="552ec3ab-d017-450f-85bb-616f65146d2e" Sep 07 16:19:53.412738 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:53.412671 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhv86" podUID="599e026d-7338-4319-a1d7-7b714bbb6e58" Sep 07 16:19:53.508717 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:53.508553 2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 07 16:19:53.557734 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:53.557645 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j6dwj" event={"ID":"c1912e17-6316-4af6-9eda-b597816af7e8","Type":"ContainerStarted","Data":"11ef9b7512b2cb4a7a72b6a36037412d41bc70932c9035f096389d4889a55647"} Sep 07 16:19:53.559371 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:53.559348 2580 generic.go:356] "Generic (PLEG): container finished" podID="2e77c59f-ef45-4fb8-99be-5fdccb594e0c" containerID="58cabab85326105658ad1d720be3d27bd670ef7f86fea65f910ba4eb69ad0050" exitCode=0 Sep 07 16:19:53.559476 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:53.559435 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7tdcz" event={"ID":"2e77c59f-ef45-4fb8-99be-5fdccb594e0c","Type":"ContainerDied","Data":"58cabab85326105658ad1d720be3d27bd670ef7f86fea65f910ba4eb69ad0050"} Sep 07 16:19:53.564577 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:53.564536 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" event={"ID":"9039f150-f93a-482e-aa48-46844ac5c7bc","Type":"ContainerStarted","Data":"0baae63dd194caae4dba6a81235371b3722812d7e3e43b8baa6659943a5ba056"} Sep 07 16:19:53.564666 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:53.564609 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" event={"ID":"9039f150-f93a-482e-aa48-46844ac5c7bc","Type":"ContainerStarted","Data":"c124beec2f04c54be54ac031ede60b2f939c11e1695883f8648a554de191a4fc"} Sep 07 16:19:53.564666 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:53.564642 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" event={"ID":"9039f150-f93a-482e-aa48-46844ac5c7bc","Type":"ContainerStarted","Data":"ee3bf09859f55d86553da257c2359ed579e18300c1b4326354d71e8d70a533a3"} Sep 07 16:19:53.564762 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:53.564676 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" event={"ID":"9039f150-f93a-482e-aa48-46844ac5c7bc","Type":"ContainerStarted","Data":"07bd80a30fa6d7eceb39e94d3562a911d77303a93b92b12714ba1387e9fb4a2e"} Sep 07 16:19:53.564762 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:53.564704 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" event={"ID":"9039f150-f93a-482e-aa48-46844ac5c7bc","Type":"ContainerStarted","Data":"ad99f36cec8718d1f6352474eb9582dc00354a7e35faef4b449dd1a7a7283c0a"} Sep 07 16:19:53.564762 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:53.564745 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" event={"ID":"9039f150-f93a-482e-aa48-46844ac5c7bc","Type":"ContainerStarted","Data":"cff0ded5ef952ba1b52ebb62e2d9aa51b7f9857a13b34d5e04377f68f9b73e6d"} Sep 07 16:19:53.565235 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:53.565141 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-hv5bm" Sep 07 16:19:53.565467 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:53.565451 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hv5bm" Sep 07 16:19:54.411518 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:54.411410 2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-07T16:19:53.508713998Z","UUID":"7259a84e-3bb1-42b0-9813-154b7691f593","Handler":null,"Name":"","Endpoint":""} Sep 07 16:19:54.413238 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:54.413217 2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 07 16:19:54.413364 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:54.413247 2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 07 16:19:54.568636 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:54.568570 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-l9vhv" event={"ID":"9dd316ae-6ec2-4359-887d-7e2665504ce4","Type":"ContainerStarted","Data":"8f6b19a4fe158f98e1b36653a28ce3ce593a6b2ae48b1b06153c88bd85658b0b"} Sep 07 16:19:54.571397 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:54.571369 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j6dwj" event={"ID":"c1912e17-6316-4af6-9eda-b597816af7e8","Type":"ContainerStarted","Data":"1bd5a5606a991ddaaa782c0b1254f7075b7973f84b82c312f86762125c0442a4"} Sep 07 16:19:54.582440 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:54.582398 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-l9vhv" podStartSLOduration=5.626967446 podStartE2EDuration="23.582386066s" totalImagesPullingTime="17.95541862s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 16:19:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:19:54.582370066 +0000 UTC m=+23.617593136" watchObservedRunningTime="2026-09-07 16:19:54.582386066 +0000 UTC m=+23.617609138" Sep 07 16:19:54.598734 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:54.598652 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j6dwj" podStartSLOduration=3.92441762 podStartE2EDuration="23.598641319s" totalImagesPullingTime="19.674223699s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 16:19:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:19:54.598579158 +0000 UTC m=+23.633802228" watchObservedRunningTime="2026-09-07 16:19:54.598641319 +0000 UTC m=+23.633864371" Sep 07 16:19:55.412919 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:55.412835 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6n92f" Sep 07 16:19:55.413093 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:55.412965 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6n92f" podUID="4f4aaec6-7ab4-48ab-a0e1-4d26977f0315" Sep 07 16:19:55.413093 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:55.413072 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8g844" Sep 07 16:19:55.413215 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:55.413192 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8g844" podUID="552ec3ab-d017-450f-85bb-616f65146d2e" Sep 07 16:19:55.413278 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:55.413234 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhv86" Sep 07 16:19:55.413365 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:55.413349 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhv86" podUID="599e026d-7338-4319-a1d7-7b714bbb6e58" Sep 07 16:19:55.580068 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:55.580014 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" event={"ID":"9039f150-f93a-482e-aa48-46844ac5c7bc","Type":"ContainerStarted","Data":"8e15564948432ce8442ab8a670aec2e3fcfefa4f5d326ad4c2e9565dcee9befa"} Sep 07 16:19:56.856044 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:56.855986 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hv5bm_4013a04f-2b59-44fa-af92-aece71a27fc8/konnectivity-agent/0.log" limit=1024 Sep 07 16:19:56.937713 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:56.937675 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hv5bm_4013a04f-2b59-44fa-af92-aece71a27fc8/konnectivity-agent/0.log" limit=1024 Sep 07 16:19:57.413073 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:57.413038 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8g844" Sep 07 16:19:57.413073 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:57.413079 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6n92f" Sep 07 16:19:57.413300 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:57.413092 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhv86" Sep 07 16:19:57.413300 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:57.413167 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8g844" podUID="552ec3ab-d017-450f-85bb-616f65146d2e" Sep 07 16:19:57.413405 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:57.413313 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6n92f" podUID="4f4aaec6-7ab4-48ab-a0e1-4d26977f0315" Sep 07 16:19:57.413462 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:57.413422 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhv86" podUID="599e026d-7338-4319-a1d7-7b714bbb6e58" Sep 07 16:19:57.834963 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:57.834922 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hv5bm_4013a04f-2b59-44fa-af92-aece71a27fc8/konnectivity-agent/0.log" limit=1024 Sep 07 16:19:57.888208 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:57.888172 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hv5bm_4013a04f-2b59-44fa-af92-aece71a27fc8/konnectivity-agent/0.log" limit=1024 Sep 07 16:19:58.589956 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:58.589777 2580 generic.go:356] "Generic (PLEG): container finished" podID="2e77c59f-ef45-4fb8-99be-5fdccb594e0c" containerID="d609ea2f88c37f1b0b3b945ed081fcef435006ac2085793e0cb039e8b4a87cb2" exitCode=0 Sep 07 16:19:58.590121 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:58.589861 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7tdcz" event={"ID":"2e77c59f-ef45-4fb8-99be-5fdccb594e0c","Type":"ContainerDied","Data":"d609ea2f88c37f1b0b3b945ed081fcef435006ac2085793e0cb039e8b4a87cb2"} Sep 07 16:19:58.593998 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:58.593975 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" event={"ID":"9039f150-f93a-482e-aa48-46844ac5c7bc","Type":"ContainerStarted","Data":"a8e96faa1d83d6fc700db61da158e9271ae3c7e1344c0ec80e513d0e2fc810fe"} Sep 07 16:19:58.594273 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:58.594256 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:58.610318 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:58.610295 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:58.627773 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:58.627731 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" podStartSLOduration=9.230274311 podStartE2EDuration="27.627715804s" totalImagesPullingTime="18.397441493s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 16:19:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:19:58.627379568 +0000 UTC m=+27.662602664" watchObservedRunningTime="2026-09-07 16:19:58.627715804 +0000 UTC m=+27.662938874" Sep 07 16:19:59.412862 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:59.412838 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhv86" Sep 07 16:19:59.413467 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:59.412838 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8g844" Sep 07 16:19:59.413467 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:59.412954 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhv86" podUID="599e026d-7338-4319-a1d7-7b714bbb6e58" Sep 07 16:19:59.413467 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:59.412838 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6n92f" Sep 07 16:19:59.413467 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:59.413061 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8g844" podUID="552ec3ab-d017-450f-85bb-616f65146d2e" Sep 07 16:19:59.413467 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:59.413143 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6n92f" podUID="4f4aaec6-7ab4-48ab-a0e1-4d26977f0315" Sep 07 16:19:59.599454 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:59.599369 2580 generic.go:356] "Generic (PLEG): container finished" podID="2e77c59f-ef45-4fb8-99be-5fdccb594e0c" containerID="1459f60813bcaa47d96afba25583a63acc6f562822adc77d2e2eafa3c370d4d4" exitCode=0 Sep 07 16:19:59.599583 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:59.599456 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7tdcz" event={"ID":"2e77c59f-ef45-4fb8-99be-5fdccb594e0c","Type":"ContainerDied","Data":"1459f60813bcaa47d96afba25583a63acc6f562822adc77d2e2eafa3c370d4d4"} Sep 07 16:19:59.599583 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:59.599511 2580 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 07 16:19:59.599997 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:59.599915 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:59.616556 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:59.616533 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:19:59.970557 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:59.970270 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6n92f"] Sep 07 16:19:59.970682 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:59.970646 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6n92f" Sep 07 16:19:59.970796 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:59.970770 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6n92f" podUID="4f4aaec6-7ab4-48ab-a0e1-4d26977f0315" Sep 07 16:19:59.972841 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:59.972807 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8g844"] Sep 07 16:19:59.972962 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:59.972906 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8g844" Sep 07 16:19:59.973044 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:59.972988 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8g844" podUID="552ec3ab-d017-450f-85bb-616f65146d2e" Sep 07 16:19:59.974145 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:59.974106 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lhv86"] Sep 07 16:19:59.974255 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:19:59.974204 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhv86" Sep 07 16:19:59.974316 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:19:59.974287 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhv86" podUID="599e026d-7338-4319-a1d7-7b714bbb6e58" Sep 07 16:20:00.603703 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:00.603612 2580 generic.go:356] "Generic (PLEG): container finished" podID="2e77c59f-ef45-4fb8-99be-5fdccb594e0c" containerID="b89417eb97a39be7d6e99b9c0b44c10dfbdc4fb903a0f8f9c7b7bda973d780a7" exitCode=0 Sep 07 16:20:00.603703 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:00.603676 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7tdcz" event={"ID":"2e77c59f-ef45-4fb8-99be-5fdccb594e0c","Type":"ContainerDied","Data":"b89417eb97a39be7d6e99b9c0b44c10dfbdc4fb903a0f8f9c7b7bda973d780a7"} Sep 07 16:20:00.604225 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:00.603754 2580 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 07 16:20:01.103418 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:01.103386 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hv5bm_4013a04f-2b59-44fa-af92-aece71a27fc8/konnectivity-agent/0.log" limit=1024 Sep 07 16:20:01.203981 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:01.203937 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hv5bm_4013a04f-2b59-44fa-af92-aece71a27fc8/konnectivity-agent/0.log" limit=1024 Sep 07 16:20:01.406915 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:01.406883 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hv5bm_4013a04f-2b59-44fa-af92-aece71a27fc8/konnectivity-agent/0.log" limit=1024 Sep 07 16:20:01.413116 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:01.413091 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8g844" Sep 07 16:20:01.413253 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:01.413157 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6n92f" Sep 07 16:20:01.413253 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:01.413189 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhv86" Sep 07 16:20:01.413253 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:20:01.413225 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8g844" podUID="552ec3ab-d017-450f-85bb-616f65146d2e" Sep 07 16:20:01.413396 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:20:01.413260 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhv86" podUID="599e026d-7338-4319-a1d7-7b714bbb6e58" Sep 07 16:20:01.413396 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:20:01.413351 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6n92f" podUID="4f4aaec6-7ab4-48ab-a0e1-4d26977f0315" Sep 07 16:20:01.505727 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:01.505687 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hv5bm_4013a04f-2b59-44fa-af92-aece71a27fc8/konnectivity-agent/0.log" limit=1024 Sep 07 16:20:01.606749 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:01.606720 2580 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 07 16:20:03.357992 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:03.357954 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:20:03.358487 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:03.358274 2580 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 07 16:20:03.374282 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:03.374249 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" podUID="9039f150-f93a-482e-aa48-46844ac5c7bc" containerName="ovnkube-controller" probeResult="failure" output="" Sep 07 16:20:03.383157 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:03.383128 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" podUID="9039f150-f93a-482e-aa48-46844ac5c7bc" containerName="ovnkube-controller" probeResult="failure" output="" Sep 07 16:20:03.412508 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:03.412474 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhv86" Sep 07 16:20:03.412508 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:03.412491 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8g844" Sep 07 16:20:03.412681 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:03.412477 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6n92f" Sep 07 16:20:03.412681 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:20:03.412613 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhv86" podUID="599e026d-7338-4319-a1d7-7b714bbb6e58" Sep 07 16:20:03.412775 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:20:03.412718 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8g844" podUID="552ec3ab-d017-450f-85bb-616f65146d2e" Sep 07 16:20:03.412826 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:20:03.412773 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6n92f" podUID="4f4aaec6-7ab4-48ab-a0e1-4d26977f0315" Sep 07 16:20:05.230101 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.230004 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/599e026d-7338-4319-a1d7-7b714bbb6e58-metrics-certs\") pod \"network-metrics-daemon-lhv86\" (UID: \"599e026d-7338-4319-a1d7-7b714bbb6e58\") " pod="openshift-multus/network-metrics-daemon-lhv86" Sep 07 16:20:05.230101 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.230072 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcgl7\" (UniqueName: \"kubernetes.io/projected/552ec3ab-d017-450f-85bb-616f65146d2e-kube-api-access-kcgl7\") pod \"network-check-target-8g844\" (UID: \"552ec3ab-d017-450f-85bb-616f65146d2e\") " pod="openshift-network-diagnostics/network-check-target-8g844" Sep 07 16:20:05.230101 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.230091 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4f4aaec6-7ab4-48ab-a0e1-4d26977f0315-original-pull-secret\") pod \"global-pull-secret-syncer-6n92f\" (UID: \"4f4aaec6-7ab4-48ab-a0e1-4d26977f0315\") " pod="kube-system/global-pull-secret-syncer-6n92f" Sep 07 16:20:05.230585 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:20:05.230169 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 16:20:05.230585 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:20:05.230176 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 07 16:20:05.230585 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:20:05.230179 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 07 16:20:05.230585 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:20:05.230230 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/599e026d-7338-4319-a1d7-7b714bbb6e58-metrics-certs podName:599e026d-7338-4319-a1d7-7b714bbb6e58 nodeName:}" failed. No retries permitted until 2026-09-07 16:20:37.230213614 +0000 UTC m=+66.265436663 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/599e026d-7338-4319-a1d7-7b714bbb6e58-metrics-certs") pod "network-metrics-daemon-lhv86" (UID: "599e026d-7338-4319-a1d7-7b714bbb6e58") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 16:20:05.230585 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:20:05.230231 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 07 16:20:05.230585 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:20:05.230244 2580 projected.go:196] Error preparing data for projected volume kube-api-access-kcgl7 for pod openshift-network-diagnostics/network-check-target-8g844: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 16:20:05.230585 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:20:05.230244 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f4aaec6-7ab4-48ab-a0e1-4d26977f0315-original-pull-secret podName:4f4aaec6-7ab4-48ab-a0e1-4d26977f0315 nodeName:}" failed. No retries permitted until 2026-09-07 16:20:37.230238557 +0000 UTC m=+66.265461605 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4f4aaec6-7ab4-48ab-a0e1-4d26977f0315-original-pull-secret") pod "global-pull-secret-syncer-6n92f" (UID: "4f4aaec6-7ab4-48ab-a0e1-4d26977f0315") : object "kube-system"/"original-pull-secret" not registered Sep 07 16:20:05.230585 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:20:05.230277 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/552ec3ab-d017-450f-85bb-616f65146d2e-kube-api-access-kcgl7 podName:552ec3ab-d017-450f-85bb-616f65146d2e nodeName:}" failed. No retries permitted until 2026-09-07 16:20:37.230264669 +0000 UTC m=+66.265487717 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-kcgl7" (UniqueName: "kubernetes.io/projected/552ec3ab-d017-450f-85bb-616f65146d2e-kube-api-access-kcgl7") pod "network-check-target-8g844" (UID: "552ec3ab-d017-450f-85bb-616f65146d2e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 16:20:05.285463 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.285427 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeReady" Sep 07 16:20:05.285650 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.285571 2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 07 16:20:05.319098 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.319072 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-f5568c6f7-f5sdc"] Sep 07 16:20:05.339786 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.339752 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-l2gc8"] Sep 07 16:20:05.339960 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.339882 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f5568c6f7-f5sdc" Sep 07 16:20:05.341920 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.341897 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-qt7fz\"" Sep 07 16:20:05.341920 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.341917 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 07 16:20:05.342140 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.341962 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 07 16:20:05.342140 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.341967 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 07 16:20:05.350855 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.350832 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 07 16:20:05.362060 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.362035 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vhwvg"] Sep 07 16:20:05.362226 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.362114 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-l2gc8" Sep 07 16:20:05.364205 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.364186 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 07 16:20:05.364510 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.364496 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-c2wmp\"" Sep 07 16:20:05.364580 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.364509 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 07 16:20:05.368455 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.368385 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hv5bm_4013a04f-2b59-44fa-af92-aece71a27fc8/konnectivity-agent/0.log" limit=1024 Sep 07 16:20:05.385626 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.385606 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-f5568c6f7-f5sdc"] Sep 07 16:20:05.385719 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.385637 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vhwvg"] Sep 07 16:20:05.385719 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.385653 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-l2gc8"] Sep 07 16:20:05.385784 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.385733 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vhwvg" Sep 07 16:20:05.388131 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.388075 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-jt2mt\"" Sep 07 16:20:05.388268 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.388147 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 07 16:20:05.388268 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.388198 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 07 16:20:05.388439 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.388286 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 07 16:20:05.412998 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.412508 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhv86" Sep 07 16:20:05.412998 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.412899 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8g844" Sep 07 16:20:05.413290 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.413276 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6n92f" Sep 07 16:20:05.415606 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.415582 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 07 16:20:05.415761 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.415728 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9lqj4\"" Sep 07 16:20:05.415869 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.415860 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 07 16:20:05.416015 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.415946 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-m4f7n\"" Sep 07 16:20:05.416164 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.416142 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 07 16:20:05.416344 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.416328 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 07 16:20:05.429497 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.429359 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hv5bm_4013a04f-2b59-44fa-af92-aece71a27fc8/konnectivity-agent/0.log" limit=1024 Sep 07 16:20:05.431452 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.431430 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9-trusted-ca\") pod \"image-registry-f5568c6f7-f5sdc\" (UID: \"ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9\") " pod="openshift-image-registry/image-registry-f5568c6f7-f5sdc" Sep 07 16:20:05.431530 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.431472 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9-registry-tls\") pod \"image-registry-f5568c6f7-f5sdc\" (UID: \"ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9\") " pod="openshift-image-registry/image-registry-f5568c6f7-f5sdc" Sep 07 16:20:05.431530 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.431512 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlj5w\" (UniqueName: \"kubernetes.io/projected/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9-kube-api-access-wlj5w\") pod \"image-registry-f5568c6f7-f5sdc\" (UID: \"ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9\") " pod="openshift-image-registry/image-registry-f5568c6f7-f5sdc" Sep 07 16:20:05.431601 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.431555 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9-registry-certificates\") pod \"image-registry-f5568c6f7-f5sdc\" (UID: \"ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9\") " pod="openshift-image-registry/image-registry-f5568c6f7-f5sdc" Sep 07 16:20:05.431601 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.431583 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9-ca-trust-extracted\") pod \"image-registry-f5568c6f7-f5sdc\" (UID: \"ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9\") " pod="openshift-image-registry/image-registry-f5568c6f7-f5sdc" Sep 07 16:20:05.431661 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.431606 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9-bound-sa-token\") pod \"image-registry-f5568c6f7-f5sdc\" (UID: \"ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9\") " pod="openshift-image-registry/image-registry-f5568c6f7-f5sdc" Sep 07 16:20:05.431702 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.431661 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9-installation-pull-secrets\") pod \"image-registry-f5568c6f7-f5sdc\" (UID: \"ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9\") " pod="openshift-image-registry/image-registry-f5568c6f7-f5sdc" Sep 07 16:20:05.431736 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.431715 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9-image-registry-private-configuration\") pod \"image-registry-f5568c6f7-f5sdc\" (UID: \"ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9\") " pod="openshift-image-registry/image-registry-f5568c6f7-f5sdc" Sep 07 16:20:05.532620 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.532588 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bc83c7d-9a35-4c79-82a4-bdad68c35006-tmp-dir\") pod \"dns-default-l2gc8\" (UID: \"5bc83c7d-9a35-4c79-82a4-bdad68c35006\") " pod="openshift-dns/dns-default-l2gc8" Sep 07 16:20:05.532802 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.532634 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl74l\" (UniqueName: \"kubernetes.io/projected/88d5615b-94e6-42ad-bf64-a9e443f8a1ba-kube-api-access-gl74l\") pod \"ingress-canary-vhwvg\" (UID: \"88d5615b-94e6-42ad-bf64-a9e443f8a1ba\") " pod="openshift-ingress-canary/ingress-canary-vhwvg" Sep 07 16:20:05.532802 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.532666 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9-installation-pull-secrets\") pod \"image-registry-f5568c6f7-f5sdc\" (UID: \"ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9\") " pod="openshift-image-registry/image-registry-f5568c6f7-f5sdc" Sep 07 16:20:05.532802 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.532743 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9-image-registry-private-configuration\") pod \"image-registry-f5568c6f7-f5sdc\" (UID: \"ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9\") " pod="openshift-image-registry/image-registry-f5568c6f7-f5sdc" Sep 07 16:20:05.532802 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.532782 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9-trusted-ca\") pod \"image-registry-f5568c6f7-f5sdc\" (UID: \"ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9\") " pod="openshift-image-registry/image-registry-f5568c6f7-f5sdc" Sep 07 16:20:05.532996 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.532808 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzrc4\" (UniqueName: \"kubernetes.io/projected/5bc83c7d-9a35-4c79-82a4-bdad68c35006-kube-api-access-qzrc4\") pod \"dns-default-l2gc8\" (UID: \"5bc83c7d-9a35-4c79-82a4-bdad68c35006\") " pod="openshift-dns/dns-default-l2gc8" Sep 07 16:20:05.532996 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.532828 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88d5615b-94e6-42ad-bf64-a9e443f8a1ba-cert\") pod \"ingress-canary-vhwvg\" (UID: \"88d5615b-94e6-42ad-bf64-a9e443f8a1ba\") " pod="openshift-ingress-canary/ingress-canary-vhwvg" Sep 07 16:20:05.532996 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.532861 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9-registry-tls\") pod \"image-registry-f5568c6f7-f5sdc\" (UID: \"ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9\") " pod="openshift-image-registry/image-registry-f5568c6f7-f5sdc" Sep 07 16:20:05.532996 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.532884 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlj5w\" (UniqueName: \"kubernetes.io/projected/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9-kube-api-access-wlj5w\") pod \"image-registry-f5568c6f7-f5sdc\" (UID: \"ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9\") " pod="openshift-image-registry/image-registry-f5568c6f7-f5sdc" Sep 07 16:20:05.532996 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.532931 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9-registry-certificates\") pod \"image-registry-f5568c6f7-f5sdc\" (UID: \"ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9\") " pod="openshift-image-registry/image-registry-f5568c6f7-f5sdc" Sep 07 16:20:05.532996 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.532954 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5bc83c7d-9a35-4c79-82a4-bdad68c35006-config-volume\") pod \"dns-default-l2gc8\" (UID: \"5bc83c7d-9a35-4c79-82a4-bdad68c35006\") " pod="openshift-dns/dns-default-l2gc8" Sep 07 16:20:05.532996 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.532981 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9-ca-trust-extracted\") pod \"image-registry-f5568c6f7-f5sdc\" (UID: \"ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9\") " pod="openshift-image-registry/image-registry-f5568c6f7-f5sdc" Sep 07 16:20:05.533310 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.533005 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9-bound-sa-token\") pod \"image-registry-f5568c6f7-f5sdc\" (UID: \"ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9\") " pod="openshift-image-registry/image-registry-f5568c6f7-f5sdc" Sep 07 16:20:05.533310 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.533050 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5bc83c7d-9a35-4c79-82a4-bdad68c35006-metrics-tls\") pod \"dns-default-l2gc8\" (UID: \"5bc83c7d-9a35-4c79-82a4-bdad68c35006\") " pod="openshift-dns/dns-default-l2gc8" Sep 07 16:20:05.533310 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:20:05.533184 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 07 16:20:05.533310 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:20:05.533197 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f5568c6f7-f5sdc: secret "image-registry-tls" not found Sep 07 16:20:05.533310 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:20:05.533253 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9-registry-tls podName:ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9 nodeName:}" failed. No retries permitted until 2026-09-07 16:20:06.033231068 +0000 UTC m=+35.068454133 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9-registry-tls") pod "image-registry-f5568c6f7-f5sdc" (UID: "ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9") : secret "image-registry-tls" not found Sep 07 16:20:05.533832 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.533806 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9-ca-trust-extracted\") pod \"image-registry-f5568c6f7-f5sdc\" (UID: \"ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9\") " pod="openshift-image-registry/image-registry-f5568c6f7-f5sdc" Sep 07 16:20:05.534065 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.533892 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9-registry-certificates\") pod \"image-registry-f5568c6f7-f5sdc\" (UID: \"ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9\") " pod="openshift-image-registry/image-registry-f5568c6f7-f5sdc" Sep 07 16:20:05.534235 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.534213 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9-trusted-ca\") pod \"image-registry-f5568c6f7-f5sdc\" (UID: \"ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9\") " pod="openshift-image-registry/image-registry-f5568c6f7-f5sdc" Sep 07 16:20:05.536807 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.536787 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9-image-registry-private-configuration\") pod \"image-registry-f5568c6f7-f5sdc\" (UID: \"ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9\") " pod="openshift-image-registry/image-registry-f5568c6f7-f5sdc" Sep 07 16:20:05.536952 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.536931 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9-installation-pull-secrets\") pod \"image-registry-f5568c6f7-f5sdc\" (UID: \"ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9\") " pod="openshift-image-registry/image-registry-f5568c6f7-f5sdc" Sep 07 16:20:05.543585 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.543559 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9-bound-sa-token\") pod \"image-registry-f5568c6f7-f5sdc\" (UID: \"ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9\") " pod="openshift-image-registry/image-registry-f5568c6f7-f5sdc" Sep 07 16:20:05.543673 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.543572 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlj5w\" (UniqueName: \"kubernetes.io/projected/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9-kube-api-access-wlj5w\") pod \"image-registry-f5568c6f7-f5sdc\" (UID: \"ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9\") " pod="openshift-image-registry/image-registry-f5568c6f7-f5sdc" Sep 07 16:20:05.623103 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.623066 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7tdcz" event={"ID":"2e77c59f-ef45-4fb8-99be-5fdccb594e0c","Type":"ContainerStarted","Data":"166d9354f56d72663fda455b53a650047a4f715ae03c2d8601787c73b2f3afa2"} Sep 07 16:20:05.633815 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.633789 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzrc4\" (UniqueName: \"kubernetes.io/projected/5bc83c7d-9a35-4c79-82a4-bdad68c35006-kube-api-access-qzrc4\") pod \"dns-default-l2gc8\" (UID: \"5bc83c7d-9a35-4c79-82a4-bdad68c35006\") " pod="openshift-dns/dns-default-l2gc8" Sep 07 16:20:05.633815 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.633820 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88d5615b-94e6-42ad-bf64-a9e443f8a1ba-cert\") pod \"ingress-canary-vhwvg\" (UID: \"88d5615b-94e6-42ad-bf64-a9e443f8a1ba\") " pod="openshift-ingress-canary/ingress-canary-vhwvg" Sep 07 16:20:05.634071 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.633859 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5bc83c7d-9a35-4c79-82a4-bdad68c35006-config-volume\") pod \"dns-default-l2gc8\" (UID: \"5bc83c7d-9a35-4c79-82a4-bdad68c35006\") " pod="openshift-dns/dns-default-l2gc8" Sep 07 16:20:05.634071 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.633877 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5bc83c7d-9a35-4c79-82a4-bdad68c35006-metrics-tls\") pod \"dns-default-l2gc8\" (UID: \"5bc83c7d-9a35-4c79-82a4-bdad68c35006\") " pod="openshift-dns/dns-default-l2gc8" Sep 07 16:20:05.634071 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.633942 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bc83c7d-9a35-4c79-82a4-bdad68c35006-tmp-dir\") pod \"dns-default-l2gc8\" (UID: \"5bc83c7d-9a35-4c79-82a4-bdad68c35006\") " pod="openshift-dns/dns-default-l2gc8" Sep 07 16:20:05.634071 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:20:05.633965 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 07 16:20:05.634071 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.633979 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl74l\" (UniqueName: \"kubernetes.io/projected/88d5615b-94e6-42ad-bf64-a9e443f8a1ba-kube-api-access-gl74l\") pod \"ingress-canary-vhwvg\" (UID: \"88d5615b-94e6-42ad-bf64-a9e443f8a1ba\") " pod="openshift-ingress-canary/ingress-canary-vhwvg" Sep 07 16:20:05.634338 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:20:05.634208 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bc83c7d-9a35-4c79-82a4-bdad68c35006-metrics-tls podName:5bc83c7d-9a35-4c79-82a4-bdad68c35006 nodeName:}" failed. No retries permitted until 2026-09-07 16:20:06.134177434 +0000 UTC m=+35.169400498 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5bc83c7d-9a35-4c79-82a4-bdad68c35006-metrics-tls") pod "dns-default-l2gc8" (UID: "5bc83c7d-9a35-4c79-82a4-bdad68c35006") : secret "dns-default-metrics-tls" not found Sep 07 16:20:05.634338 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.634302 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bc83c7d-9a35-4c79-82a4-bdad68c35006-tmp-dir\") pod \"dns-default-l2gc8\" (UID: \"5bc83c7d-9a35-4c79-82a4-bdad68c35006\") " pod="openshift-dns/dns-default-l2gc8" Sep 07 16:20:05.634572 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.634549 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5bc83c7d-9a35-4c79-82a4-bdad68c35006-config-volume\") pod \"dns-default-l2gc8\" (UID: \"5bc83c7d-9a35-4c79-82a4-bdad68c35006\") " pod="openshift-dns/dns-default-l2gc8" Sep 07 16:20:05.636439 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.636422 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88d5615b-94e6-42ad-bf64-a9e443f8a1ba-cert\") pod \"ingress-canary-vhwvg\" (UID: \"88d5615b-94e6-42ad-bf64-a9e443f8a1ba\") " pod="openshift-ingress-canary/ingress-canary-vhwvg" Sep 07 16:20:05.641635 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.641608 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzrc4\" (UniqueName: \"kubernetes.io/projected/5bc83c7d-9a35-4c79-82a4-bdad68c35006-kube-api-access-qzrc4\") pod \"dns-default-l2gc8\" (UID: \"5bc83c7d-9a35-4c79-82a4-bdad68c35006\") " pod="openshift-dns/dns-default-l2gc8" Sep 07 16:20:05.641726 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.641613 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl74l\" (UniqueName: \"kubernetes.io/projected/88d5615b-94e6-42ad-bf64-a9e443f8a1ba-kube-api-access-gl74l\") pod \"ingress-canary-vhwvg\" (UID: \"88d5615b-94e6-42ad-bf64-a9e443f8a1ba\") " pod="openshift-ingress-canary/ingress-canary-vhwvg" Sep 07 16:20:05.693964 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.693928 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vhwvg" Sep 07 16:20:05.990137 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:05.990100 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vhwvg"] Sep 07 16:20:05.993730 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:20:05.993699 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88d5615b_94e6_42ad_bf64_a9e443f8a1ba.slice/crio-d66c2b29ecbda868b24994c90e3383960b1d0a81943fbbf4398021670fca1826 WatchSource:0}: Error finding container d66c2b29ecbda868b24994c90e3383960b1d0a81943fbbf4398021670fca1826: Status 404 returned error can't find the container with id d66c2b29ecbda868b24994c90e3383960b1d0a81943fbbf4398021670fca1826 Sep 07 16:20:06.037729 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:06.037642 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9-registry-tls\") pod \"image-registry-f5568c6f7-f5sdc\" (UID: \"ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9\") " pod="openshift-image-registry/image-registry-f5568c6f7-f5sdc" Sep 07 16:20:06.037905 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:20:06.037810 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 07 16:20:06.037905 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:20:06.037831 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f5568c6f7-f5sdc: secret "image-registry-tls" not found Sep 07 16:20:06.037905 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:20:06.037899 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9-registry-tls podName:ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9 nodeName:}" failed. No retries permitted until 2026-09-07 16:20:07.03788002 +0000 UTC m=+36.073103071 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9-registry-tls") pod "image-registry-f5568c6f7-f5sdc" (UID: "ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9") : secret "image-registry-tls" not found Sep 07 16:20:06.138264 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:06.138137 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5bc83c7d-9a35-4c79-82a4-bdad68c35006-metrics-tls\") pod \"dns-default-l2gc8\" (UID: \"5bc83c7d-9a35-4c79-82a4-bdad68c35006\") " pod="openshift-dns/dns-default-l2gc8" Sep 07 16:20:06.138463 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:20:06.138292 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 07 16:20:06.138463 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:20:06.138362 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bc83c7d-9a35-4c79-82a4-bdad68c35006-metrics-tls podName:5bc83c7d-9a35-4c79-82a4-bdad68c35006 nodeName:}" failed. No retries permitted until 2026-09-07 16:20:07.138346084 +0000 UTC m=+36.173569136 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5bc83c7d-9a35-4c79-82a4-bdad68c35006-metrics-tls") pod "dns-default-l2gc8" (UID: "5bc83c7d-9a35-4c79-82a4-bdad68c35006") : secret "dns-default-metrics-tls" not found Sep 07 16:20:06.629687 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:06.629647 2580 generic.go:356] "Generic (PLEG): container finished" podID="2e77c59f-ef45-4fb8-99be-5fdccb594e0c" containerID="166d9354f56d72663fda455b53a650047a4f715ae03c2d8601787c73b2f3afa2" exitCode=0 Sep 07 16:20:06.630623 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:06.629744 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7tdcz" event={"ID":"2e77c59f-ef45-4fb8-99be-5fdccb594e0c","Type":"ContainerDied","Data":"166d9354f56d72663fda455b53a650047a4f715ae03c2d8601787c73b2f3afa2"} Sep 07 16:20:06.631404 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:06.631379 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vhwvg" event={"ID":"88d5615b-94e6-42ad-bf64-a9e443f8a1ba","Type":"ContainerStarted","Data":"d66c2b29ecbda868b24994c90e3383960b1d0a81943fbbf4398021670fca1826"} Sep 07 16:20:06.650956 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:06.650921 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hv5bm_4013a04f-2b59-44fa-af92-aece71a27fc8/konnectivity-agent/0.log" limit=1024 Sep 07 16:20:06.704279 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:06.703574 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hv5bm_4013a04f-2b59-44fa-af92-aece71a27fc8/konnectivity-agent/0.log" limit=1024 Sep 07 16:20:07.045603 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:07.045571 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9-registry-tls\") pod \"image-registry-f5568c6f7-f5sdc\" (UID: \"ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9\") " pod="openshift-image-registry/image-registry-f5568c6f7-f5sdc" Sep 07 16:20:07.045817 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:20:07.045749 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 07 16:20:07.045817 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:20:07.045767 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f5568c6f7-f5sdc: secret "image-registry-tls" not found Sep 07 16:20:07.046016 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:20:07.045825 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9-registry-tls podName:ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9 nodeName:}" failed. No retries permitted until 2026-09-07 16:20:09.045809383 +0000 UTC m=+38.081032432 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9-registry-tls") pod "image-registry-f5568c6f7-f5sdc" (UID: "ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9") : secret "image-registry-tls" not found Sep 07 16:20:07.146040 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:07.145977 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5bc83c7d-9a35-4c79-82a4-bdad68c35006-metrics-tls\") pod \"dns-default-l2gc8\" (UID: \"5bc83c7d-9a35-4c79-82a4-bdad68c35006\") " pod="openshift-dns/dns-default-l2gc8" Sep 07 16:20:07.146228 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:20:07.146159 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 07 16:20:07.146288 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:20:07.146226 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bc83c7d-9a35-4c79-82a4-bdad68c35006-metrics-tls podName:5bc83c7d-9a35-4c79-82a4-bdad68c35006 nodeName:}" failed. No retries permitted until 2026-09-07 16:20:09.146209495 +0000 UTC m=+38.181432543 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5bc83c7d-9a35-4c79-82a4-bdad68c35006-metrics-tls") pod "dns-default-l2gc8" (UID: "5bc83c7d-9a35-4c79-82a4-bdad68c35006") : secret "dns-default-metrics-tls" not found Sep 07 16:20:07.337082 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:07.336983 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hv5bm_4013a04f-2b59-44fa-af92-aece71a27fc8/konnectivity-agent/0.log" limit=1024 Sep 07 16:20:07.385177 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:07.385143 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hv5bm_4013a04f-2b59-44fa-af92-aece71a27fc8/konnectivity-agent/0.log" limit=1024 Sep 07 16:20:07.638003 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:07.637910 2580 generic.go:356] "Generic (PLEG): container finished" podID="2e77c59f-ef45-4fb8-99be-5fdccb594e0c" containerID="a27ab8f9bc4c0bb755e44094d803de1b7338c99798ef2f27aa76abf10c23cda7" exitCode=0 Sep 07 16:20:07.638003 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:07.637995 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7tdcz" event={"ID":"2e77c59f-ef45-4fb8-99be-5fdccb594e0c","Type":"ContainerDied","Data":"a27ab8f9bc4c0bb755e44094d803de1b7338c99798ef2f27aa76abf10c23cda7"} Sep 07 16:20:07.940442 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:07.940349 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-kwckr"] Sep 07 16:20:07.947165 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:07.947127 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hv5bm_4013a04f-2b59-44fa-af92-aece71a27fc8/konnectivity-agent/0.log" limit=1024 Sep 07 16:20:07.964909 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:07.964869 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kwckr"] Sep 07 16:20:07.965115 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:07.965001 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kwckr" Sep 07 16:20:07.967608 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:07.967580 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 07 16:20:07.968149 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:07.968125 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-s7rhg\"" Sep 07 16:20:07.968254 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:07.968156 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 07 16:20:07.968387 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:07.968371 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 07 16:20:07.968387 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:07.968380 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 07 16:20:08.003337 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:08.003111 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hv5bm_4013a04f-2b59-44fa-af92-aece71a27fc8/konnectivity-agent/0.log" limit=1024 Sep 07 16:20:08.155152 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:08.155114 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/89ae6799-1f6e-422c-82b3-ab64bf3967b4-data-volume\") pod \"insights-runtime-extractor-kwckr\" (UID: \"89ae6799-1f6e-422c-82b3-ab64bf3967b4\") " pod="openshift-insights/insights-runtime-extractor-kwckr" Sep 07 16:20:08.155329 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:08.155165 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/89ae6799-1f6e-422c-82b3-ab64bf3967b4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kwckr\" (UID: \"89ae6799-1f6e-422c-82b3-ab64bf3967b4\") " pod="openshift-insights/insights-runtime-extractor-kwckr" Sep 07 16:20:08.155329 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:08.155279 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/89ae6799-1f6e-422c-82b3-ab64bf3967b4-crio-socket\") pod \"insights-runtime-extractor-kwckr\" (UID: \"89ae6799-1f6e-422c-82b3-ab64bf3967b4\") " pod="openshift-insights/insights-runtime-extractor-kwckr" Sep 07 16:20:08.155431 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:08.155387 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhrmh\" (UniqueName: \"kubernetes.io/projected/89ae6799-1f6e-422c-82b3-ab64bf3967b4-kube-api-access-rhrmh\") pod \"insights-runtime-extractor-kwckr\" (UID: \"89ae6799-1f6e-422c-82b3-ab64bf3967b4\") " pod="openshift-insights/insights-runtime-extractor-kwckr" Sep 07 16:20:08.155431 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:08.155424 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/89ae6799-1f6e-422c-82b3-ab64bf3967b4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kwckr\" (UID: \"89ae6799-1f6e-422c-82b3-ab64bf3967b4\") " pod="openshift-insights/insights-runtime-extractor-kwckr" Sep 07 16:20:08.256488 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:08.256447 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhrmh\" (UniqueName: \"kubernetes.io/projected/89ae6799-1f6e-422c-82b3-ab64bf3967b4-kube-api-access-rhrmh\") pod \"insights-runtime-extractor-kwckr\" (UID: \"89ae6799-1f6e-422c-82b3-ab64bf3967b4\") " pod="openshift-insights/insights-runtime-extractor-kwckr" Sep 07 16:20:08.256691 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:08.256505 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/89ae6799-1f6e-422c-82b3-ab64bf3967b4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kwckr\" (UID: \"89ae6799-1f6e-422c-82b3-ab64bf3967b4\") " pod="openshift-insights/insights-runtime-extractor-kwckr" Sep 07 16:20:08.256691 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:08.256536 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/89ae6799-1f6e-422c-82b3-ab64bf3967b4-data-volume\") pod \"insights-runtime-extractor-kwckr\" (UID: \"89ae6799-1f6e-422c-82b3-ab64bf3967b4\") " pod="openshift-insights/insights-runtime-extractor-kwckr" Sep 07 16:20:08.256691 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:08.256575 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/89ae6799-1f6e-422c-82b3-ab64bf3967b4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kwckr\" (UID: \"89ae6799-1f6e-422c-82b3-ab64bf3967b4\") " pod="openshift-insights/insights-runtime-extractor-kwckr" Sep 07 16:20:08.256691 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:08.256638 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/89ae6799-1f6e-422c-82b3-ab64bf3967b4-crio-socket\") pod \"insights-runtime-extractor-kwckr\" (UID: \"89ae6799-1f6e-422c-82b3-ab64bf3967b4\") " pod="openshift-insights/insights-runtime-extractor-kwckr" Sep 07 16:20:08.256691 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:20:08.256649 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 07 16:20:08.256916 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:20:08.256729 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/89ae6799-1f6e-422c-82b3-ab64bf3967b4-kube-rbac-proxy-cm podName:89ae6799-1f6e-422c-82b3-ab64bf3967b4 nodeName:}" failed. No retries permitted until 2026-09-07 16:20:08.756707258 +0000 UTC m=+37.791930357 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/89ae6799-1f6e-422c-82b3-ab64bf3967b4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kwckr" (UID: "89ae6799-1f6e-422c-82b3-ab64bf3967b4") : configmap "kube-rbac-proxy" not found Sep 07 16:20:08.256916 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:20:08.256771 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 07 16:20:08.256916 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:20:08.256812 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89ae6799-1f6e-422c-82b3-ab64bf3967b4-insights-runtime-extractor-tls podName:89ae6799-1f6e-422c-82b3-ab64bf3967b4 nodeName:}" failed. No retries permitted until 2026-09-07 16:20:08.756799172 +0000 UTC m=+37.792022224 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/89ae6799-1f6e-422c-82b3-ab64bf3967b4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kwckr" (UID: "89ae6799-1f6e-422c-82b3-ab64bf3967b4") : secret "insights-runtime-extractor-tls" not found Sep 07 16:20:08.256916 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:08.256881 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/89ae6799-1f6e-422c-82b3-ab64bf3967b4-data-volume\") pod \"insights-runtime-extractor-kwckr\" (UID: \"89ae6799-1f6e-422c-82b3-ab64bf3967b4\") " pod="openshift-insights/insights-runtime-extractor-kwckr" Sep 07 16:20:08.256916 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:08.256901 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/89ae6799-1f6e-422c-82b3-ab64bf3967b4-crio-socket\") pod \"insights-runtime-extractor-kwckr\" (UID: \"89ae6799-1f6e-422c-82b3-ab64bf3967b4\") " pod="openshift-insights/insights-runtime-extractor-kwckr" Sep 07 16:20:08.265337 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:08.265308 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhrmh\" (UniqueName: \"kubernetes.io/projected/89ae6799-1f6e-422c-82b3-ab64bf3967b4-kube-api-access-rhrmh\") pod \"insights-runtime-extractor-kwckr\" (UID: \"89ae6799-1f6e-422c-82b3-ab64bf3967b4\") " pod="openshift-insights/insights-runtime-extractor-kwckr" Sep 07 16:20:08.643394 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:08.643342 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hv5bm_4013a04f-2b59-44fa-af92-aece71a27fc8/konnectivity-agent/0.log" limit=1024 Sep 07 16:20:08.644454 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:08.644339 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7tdcz" event={"ID":"2e77c59f-ef45-4fb8-99be-5fdccb594e0c","Type":"ContainerStarted","Data":"bd4d2f1de9d70f72fd491e5d6c93bb0a4154fc22cbbe1a835f270e3a7d440ee3"} Sep 07 16:20:08.667091 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:08.667049 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-7tdcz" podStartSLOduration=9.20290225 podStartE2EDuration="37.667009823s" totalImagesPullingTime="28.464107573s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 16:19:31 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:20:08.666226331 +0000 UTC m=+37.701449402" watchObservedRunningTime="2026-09-07 16:20:08.667009823 +0000 UTC m=+37.702232894" Sep 07 16:20:08.705430 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:08.705404 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fxdvm"] Sep 07 16:20:08.723570 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:08.723542 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fxdvm"] Sep 07 16:20:08.723666 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:08.723655 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fxdvm" Sep 07 16:20:08.726668 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:08.726466 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Sep 07 16:20:08.728628 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:08.728610 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-n2hjr\"" Sep 07 16:20:08.730825 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:08.730805 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Sep 07 16:20:08.749967 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:08.749940 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hv5bm_4013a04f-2b59-44fa-af92-aece71a27fc8/konnectivity-agent/0.log" limit=1024 Sep 07 16:20:08.761908 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:08.761884 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/89ae6799-1f6e-422c-82b3-ab64bf3967b4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kwckr\" (UID: \"89ae6799-1f6e-422c-82b3-ab64bf3967b4\") " pod="openshift-insights/insights-runtime-extractor-kwckr" Sep 07 16:20:08.762006 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:08.761922 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/89ae6799-1f6e-422c-82b3-ab64bf3967b4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kwckr\" (UID: \"89ae6799-1f6e-422c-82b3-ab64bf3967b4\") " pod="openshift-insights/insights-runtime-extractor-kwckr" Sep 07 16:20:08.762084 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:20:08.762037 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 07 16:20:08.762139 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:20:08.762087 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89ae6799-1f6e-422c-82b3-ab64bf3967b4-insights-runtime-extractor-tls podName:89ae6799-1f6e-422c-82b3-ab64bf3967b4 nodeName:}" failed. No retries permitted until 2026-09-07 16:20:09.762074539 +0000 UTC m=+38.797297588 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/89ae6799-1f6e-422c-82b3-ab64bf3967b4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kwckr" (UID: "89ae6799-1f6e-422c-82b3-ab64bf3967b4") : secret "insights-runtime-extractor-tls" not found Sep 07 16:20:08.762204 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:20:08.762191 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 07 16:20:08.762250 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:20:08.762238 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/89ae6799-1f6e-422c-82b3-ab64bf3967b4-kube-rbac-proxy-cm podName:89ae6799-1f6e-422c-82b3-ab64bf3967b4 nodeName:}" failed. No retries permitted until 2026-09-07 16:20:09.762224674 +0000 UTC m=+38.797447744 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/89ae6799-1f6e-422c-82b3-ab64bf3967b4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kwckr" (UID: "89ae6799-1f6e-422c-82b3-ab64bf3967b4") : configmap "kube-rbac-proxy" not found Sep 07 16:20:08.840781 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:08.840747 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hv5bm_4013a04f-2b59-44fa-af92-aece71a27fc8/konnectivity-agent/0.log" limit=1024 Sep 07 16:20:08.862784 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:08.862701 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpcdh\" (UniqueName: \"kubernetes.io/projected/509635fb-d929-4bbd-bdfc-f875c2771bf6-kube-api-access-lpcdh\") pod \"migrator-fdbb8fdff-fxdvm\" (UID: \"509635fb-d929-4bbd-bdfc-f875c2771bf6\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fxdvm" Sep 07 16:20:08.900132 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:08.900105 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hv5bm_4013a04f-2b59-44fa-af92-aece71a27fc8/konnectivity-agent/0.log" limit=1024 Sep 07 16:20:08.964223 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:08.964187 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpcdh\" (UniqueName: \"kubernetes.io/projected/509635fb-d929-4bbd-bdfc-f875c2771bf6-kube-api-access-lpcdh\") pod \"migrator-fdbb8fdff-fxdvm\" (UID: \"509635fb-d929-4bbd-bdfc-f875c2771bf6\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fxdvm" Sep 07 16:20:08.972952 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:08.972922 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpcdh\" (UniqueName: \"kubernetes.io/projected/509635fb-d929-4bbd-bdfc-f875c2771bf6-kube-api-access-lpcdh\") pod \"migrator-fdbb8fdff-fxdvm\" (UID: \"509635fb-d929-4bbd-bdfc-f875c2771bf6\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fxdvm" Sep 07 16:20:09.032959 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:09.032925 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fxdvm" Sep 07 16:20:09.065475 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:09.065444 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9-registry-tls\") pod \"image-registry-f5568c6f7-f5sdc\" (UID: \"ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9\") " pod="openshift-image-registry/image-registry-f5568c6f7-f5sdc" Sep 07 16:20:09.065616 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:20:09.065596 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 07 16:20:09.065663 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:20:09.065619 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f5568c6f7-f5sdc: secret "image-registry-tls" not found Sep 07 16:20:09.065696 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:20:09.065675 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9-registry-tls podName:ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9 nodeName:}" failed. No retries permitted until 2026-09-07 16:20:13.065657519 +0000 UTC m=+42.100880568 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9-registry-tls") pod "image-registry-f5568c6f7-f5sdc" (UID: "ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9") : secret "image-registry-tls" not found Sep 07 16:20:09.166613 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:09.166577 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5bc83c7d-9a35-4c79-82a4-bdad68c35006-metrics-tls\") pod \"dns-default-l2gc8\" (UID: \"5bc83c7d-9a35-4c79-82a4-bdad68c35006\") " pod="openshift-dns/dns-default-l2gc8" Sep 07 16:20:09.166767 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:20:09.166689 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 07 16:20:09.166767 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:20:09.166742 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bc83c7d-9a35-4c79-82a4-bdad68c35006-metrics-tls podName:5bc83c7d-9a35-4c79-82a4-bdad68c35006 nodeName:}" failed. No retries permitted until 2026-09-07 16:20:13.166729123 +0000 UTC m=+42.201952172 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5bc83c7d-9a35-4c79-82a4-bdad68c35006-metrics-tls") pod "dns-default-l2gc8" (UID: "5bc83c7d-9a35-4c79-82a4-bdad68c35006") : secret "dns-default-metrics-tls" not found Sep 07 16:20:09.330844 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:09.330812 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hv5bm_4013a04f-2b59-44fa-af92-aece71a27fc8/konnectivity-agent/0.log" limit=1024 Sep 07 16:20:09.341403 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:09.341357 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fxdvm"] Sep 07 16:20:09.346608 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:20:09.346583 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod509635fb_d929_4bbd_bdfc_f875c2771bf6.slice/crio-211ef9f9882d41869d62190b2459f45d7d17f6e9e415bb199eda12f9bb64bd9b WatchSource:0}: Error finding container 211ef9f9882d41869d62190b2459f45d7d17f6e9e415bb199eda12f9bb64bd9b: Status 404 returned error can't find the container with id 211ef9f9882d41869d62190b2459f45d7d17f6e9e415bb199eda12f9bb64bd9b Sep 07 16:20:09.431265 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:09.430984 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hv5bm_4013a04f-2b59-44fa-af92-aece71a27fc8/konnectivity-agent/0.log" limit=1024 Sep 07 16:20:09.601389 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:09.601366 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kcb7q_553ed3c9-3691-4532-8a44-702ea20b17a7/dns-node-resolver/0.log" Sep 07 16:20:09.648772 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:09.648740 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vhwvg" event={"ID":"88d5615b-94e6-42ad-bf64-a9e443f8a1ba","Type":"ContainerStarted","Data":"2aca25d93eb6be855e2d059def3ae539b36f22f78030a9145287c571fc2c4531"} Sep 07 16:20:09.649987 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:09.649962 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fxdvm" event={"ID":"509635fb-d929-4bbd-bdfc-f875c2771bf6","Type":"ContainerStarted","Data":"211ef9f9882d41869d62190b2459f45d7d17f6e9e415bb199eda12f9bb64bd9b"} Sep 07 16:20:09.664153 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:09.664105 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vhwvg" podStartSLOduration=2.049691111 podStartE2EDuration="4.664089508s" totalImagesPullingTime="2.614398397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:20:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:20:09.663770557 +0000 UTC m=+38.698993628" watchObservedRunningTime="2026-09-07 16:20:09.664089508 +0000 UTC m=+38.699312588" Sep 07 16:20:09.771613 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:09.771575 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/89ae6799-1f6e-422c-82b3-ab64bf3967b4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kwckr\" (UID: \"89ae6799-1f6e-422c-82b3-ab64bf3967b4\") " pod="openshift-insights/insights-runtime-extractor-kwckr" Sep 07 16:20:09.771813 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:09.771626 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/89ae6799-1f6e-422c-82b3-ab64bf3967b4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kwckr\" (UID: \"89ae6799-1f6e-422c-82b3-ab64bf3967b4\") " pod="openshift-insights/insights-runtime-extractor-kwckr" Sep 07 16:20:09.771813 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:20:09.771686 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 07 16:20:09.771813 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:20:09.771752 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 07 16:20:09.771813 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:20:09.771756 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/89ae6799-1f6e-422c-82b3-ab64bf3967b4-kube-rbac-proxy-cm podName:89ae6799-1f6e-422c-82b3-ab64bf3967b4 nodeName:}" failed. No retries permitted until 2026-09-07 16:20:11.771735982 +0000 UTC m=+40.806959048 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/89ae6799-1f6e-422c-82b3-ab64bf3967b4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kwckr" (UID: "89ae6799-1f6e-422c-82b3-ab64bf3967b4") : configmap "kube-rbac-proxy" not found Sep 07 16:20:09.771813 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:20:09.771799 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89ae6799-1f6e-422c-82b3-ab64bf3967b4-insights-runtime-extractor-tls podName:89ae6799-1f6e-422c-82b3-ab64bf3967b4 nodeName:}" failed. No retries permitted until 2026-09-07 16:20:11.771786367 +0000 UTC m=+40.807009415 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/89ae6799-1f6e-422c-82b3-ab64bf3967b4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kwckr" (UID: "89ae6799-1f6e-422c-82b3-ab64bf3967b4") : secret "insights-runtime-extractor-tls" not found Sep 07 16:20:10.402241 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:10.402209 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-g9wgk_8c534a46-aee8-4f33-a447-c049fb3d3f04/node-ca/0.log" Sep 07 16:20:10.494563 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:10.494530 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hv5bm_4013a04f-2b59-44fa-af92-aece71a27fc8/konnectivity-agent/0.log" limit=1024 Sep 07 16:20:10.539790 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:10.539738 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hv5bm_4013a04f-2b59-44fa-af92-aece71a27fc8/konnectivity-agent/0.log" limit=1024 Sep 07 16:20:10.929514 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:10.929479 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-7v9h2"] Sep 07 16:20:10.958572 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:10.958541 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-7v9h2"] Sep 07 16:20:10.958713 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:10.958631 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-7v9h2" Sep 07 16:20:10.961169 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:10.961147 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Sep 07 16:20:10.961279 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:10.961191 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Sep 07 16:20:10.961687 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:10.961673 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Sep 07 16:20:10.961890 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:10.961878 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Sep 07 16:20:10.961991 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:10.961976 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-v2css\"" Sep 07 16:20:11.033663 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:11.033619 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hv5bm_4013a04f-2b59-44fa-af92-aece71a27fc8/konnectivity-agent/0.log" limit=1024 Sep 07 16:20:11.081294 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:11.081262 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrntg\" (UniqueName: \"kubernetes.io/projected/5d7537a7-6a94-4b85-b330-272e4cae2a70-kube-api-access-xrntg\") pod \"service-ca-75d99c484-7v9h2\" (UID: \"5d7537a7-6a94-4b85-b330-272e4cae2a70\") " pod="openshift-service-ca/service-ca-75d99c484-7v9h2" Sep 07 16:20:11.081507 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:11.081330 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/5d7537a7-6a94-4b85-b330-272e4cae2a70-signing-key\") pod \"service-ca-75d99c484-7v9h2\" (UID: \"5d7537a7-6a94-4b85-b330-272e4cae2a70\") " pod="openshift-service-ca/service-ca-75d99c484-7v9h2" Sep 07 16:20:11.081507 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:11.081396 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/5d7537a7-6a94-4b85-b330-272e4cae2a70-signing-cabundle\") pod \"service-ca-75d99c484-7v9h2\" (UID: \"5d7537a7-6a94-4b85-b330-272e4cae2a70\") " pod="openshift-service-ca/service-ca-75d99c484-7v9h2" Sep 07 16:20:11.084981 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:11.084953 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hv5bm_4013a04f-2b59-44fa-af92-aece71a27fc8/konnectivity-agent/0.log" limit=1024 Sep 07 16:20:11.182357 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:11.182274 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrntg\" (UniqueName: \"kubernetes.io/projected/5d7537a7-6a94-4b85-b330-272e4cae2a70-kube-api-access-xrntg\") pod \"service-ca-75d99c484-7v9h2\" (UID: \"5d7537a7-6a94-4b85-b330-272e4cae2a70\") " pod="openshift-service-ca/service-ca-75d99c484-7v9h2" Sep 07 16:20:11.182519 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:11.182371 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/5d7537a7-6a94-4b85-b330-272e4cae2a70-signing-key\") pod \"service-ca-75d99c484-7v9h2\" (UID: \"5d7537a7-6a94-4b85-b330-272e4cae2a70\") " pod="openshift-service-ca/service-ca-75d99c484-7v9h2" Sep 07 16:20:11.182519 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:11.182396 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/5d7537a7-6a94-4b85-b330-272e4cae2a70-signing-cabundle\") pod \"service-ca-75d99c484-7v9h2\" (UID: \"5d7537a7-6a94-4b85-b330-272e4cae2a70\") " pod="openshift-service-ca/service-ca-75d99c484-7v9h2" Sep 07 16:20:11.183067 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:11.183046 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/5d7537a7-6a94-4b85-b330-272e4cae2a70-signing-cabundle\") pod \"service-ca-75d99c484-7v9h2\" (UID: \"5d7537a7-6a94-4b85-b330-272e4cae2a70\") " pod="openshift-service-ca/service-ca-75d99c484-7v9h2" Sep 07 16:20:11.185877 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:11.185856 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/5d7537a7-6a94-4b85-b330-272e4cae2a70-signing-key\") pod \"service-ca-75d99c484-7v9h2\" (UID: \"5d7537a7-6a94-4b85-b330-272e4cae2a70\") " pod="openshift-service-ca/service-ca-75d99c484-7v9h2" Sep 07 16:20:11.189738 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:11.189712 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrntg\" (UniqueName: \"kubernetes.io/projected/5d7537a7-6a94-4b85-b330-272e4cae2a70-kube-api-access-xrntg\") pod \"service-ca-75d99c484-7v9h2\" (UID: \"5d7537a7-6a94-4b85-b330-272e4cae2a70\") " pod="openshift-service-ca/service-ca-75d99c484-7v9h2" Sep 07 16:20:11.267668 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:11.267628 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-7v9h2" Sep 07 16:20:11.404245 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:11.404219 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vhwvg_88d5615b-94e6-42ad-bf64-a9e443f8a1ba/serve-healthcheck-canary/0.log" Sep 07 16:20:11.603104 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:11.603068 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-7v9h2"] Sep 07 16:20:11.607158 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:20:11.607131 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d7537a7_6a94_4b85_b330_272e4cae2a70.slice/crio-0814e8ab827a630016d1f8825c36a4a1fa0916010d43631c116306597de2067c WatchSource:0}: Error finding container 0814e8ab827a630016d1f8825c36a4a1fa0916010d43631c116306597de2067c: Status 404 returned error can't find the container with id 0814e8ab827a630016d1f8825c36a4a1fa0916010d43631c116306597de2067c Sep 07 16:20:11.656335 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:11.656297 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fxdvm" event={"ID":"509635fb-d929-4bbd-bdfc-f875c2771bf6","Type":"ContainerStarted","Data":"5b46d834386816987bd77790711d358ca79e7f7baef7220ef44d8b291d9846d6"} Sep 07 16:20:11.656514 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:11.656365 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fxdvm" event={"ID":"509635fb-d929-4bbd-bdfc-f875c2771bf6","Type":"ContainerStarted","Data":"f90a4c2432638e43bad1894895acdb7d8afbe980b9bfc1d8c30d725b0d7a7002"} Sep 07 16:20:11.657435 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:11.657412 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-7v9h2" event={"ID":"5d7537a7-6a94-4b85-b330-272e4cae2a70","Type":"ContainerStarted","Data":"0814e8ab827a630016d1f8825c36a4a1fa0916010d43631c116306597de2067c"} Sep 07 16:20:11.674191 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:11.674144 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fxdvm" podStartSLOduration=1.685469718 podStartE2EDuration="3.674128264s" totalImagesPullingTime="1.988658546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:20:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:20:11.673348211 +0000 UTC m=+40.708571283" watchObservedRunningTime="2026-09-07 16:20:11.674128264 +0000 UTC m=+40.709351368" Sep 07 16:20:11.787524 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:11.787489 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/89ae6799-1f6e-422c-82b3-ab64bf3967b4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kwckr\" (UID: \"89ae6799-1f6e-422c-82b3-ab64bf3967b4\") " pod="openshift-insights/insights-runtime-extractor-kwckr" Sep 07 16:20:11.787524 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:11.787528 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/89ae6799-1f6e-422c-82b3-ab64bf3967b4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kwckr\" (UID: \"89ae6799-1f6e-422c-82b3-ab64bf3967b4\") " pod="openshift-insights/insights-runtime-extractor-kwckr" Sep 07 16:20:11.787738 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:20:11.787615 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 07 16:20:11.787738 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:20:11.787676 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 07 16:20:11.787738 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:20:11.787685 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/89ae6799-1f6e-422c-82b3-ab64bf3967b4-kube-rbac-proxy-cm podName:89ae6799-1f6e-422c-82b3-ab64bf3967b4 nodeName:}" failed. No retries permitted until 2026-09-07 16:20:15.787664541 +0000 UTC m=+44.822887590 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/89ae6799-1f6e-422c-82b3-ab64bf3967b4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kwckr" (UID: "89ae6799-1f6e-422c-82b3-ab64bf3967b4") : configmap "kube-rbac-proxy" not found Sep 07 16:20:11.787738 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:20:11.787711 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89ae6799-1f6e-422c-82b3-ab64bf3967b4-insights-runtime-extractor-tls podName:89ae6799-1f6e-422c-82b3-ab64bf3967b4 nodeName:}" failed. No retries permitted until 2026-09-07 16:20:15.787701703 +0000 UTC m=+44.822924767 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/89ae6799-1f6e-422c-82b3-ab64bf3967b4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kwckr" (UID: "89ae6799-1f6e-422c-82b3-ab64bf3967b4") : secret "insights-runtime-extractor-tls" not found Sep 07 16:20:11.796534 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:11.796499 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hv5bm_4013a04f-2b59-44fa-af92-aece71a27fc8/konnectivity-agent/0.log" limit=1024 Sep 07 16:20:12.721075 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:12.720999 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hv5bm_4013a04f-2b59-44fa-af92-aece71a27fc8/konnectivity-agent/0.log" limit=1024 Sep 07 16:20:13.099366 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:13.099285 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9-registry-tls\") pod \"image-registry-f5568c6f7-f5sdc\" (UID: \"ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9\") " pod="openshift-image-registry/image-registry-f5568c6f7-f5sdc" Sep 07 16:20:13.099504 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:20:13.099460 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 07 16:20:13.099504 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:20:13.099479 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f5568c6f7-f5sdc: secret "image-registry-tls" not found Sep 07 16:20:13.099575 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:20:13.099537 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9-registry-tls podName:ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9 nodeName:}" failed. No retries permitted until 2026-09-07 16:20:21.099519492 +0000 UTC m=+50.134742540 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9-registry-tls") pod "image-registry-f5568c6f7-f5sdc" (UID: "ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9") : secret "image-registry-tls" not found Sep 07 16:20:13.200678 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:13.200646 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5bc83c7d-9a35-4c79-82a4-bdad68c35006-metrics-tls\") pod \"dns-default-l2gc8\" (UID: \"5bc83c7d-9a35-4c79-82a4-bdad68c35006\") " pod="openshift-dns/dns-default-l2gc8" Sep 07 16:20:13.200831 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:20:13.200807 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 07 16:20:13.200886 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:20:13.200875 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bc83c7d-9a35-4c79-82a4-bdad68c35006-metrics-tls podName:5bc83c7d-9a35-4c79-82a4-bdad68c35006 nodeName:}" failed. No retries permitted until 2026-09-07 16:20:21.200858224 +0000 UTC m=+50.236081273 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5bc83c7d-9a35-4c79-82a4-bdad68c35006-metrics-tls") pod "dns-default-l2gc8" (UID: "5bc83c7d-9a35-4c79-82a4-bdad68c35006") : secret "dns-default-metrics-tls" not found Sep 07 16:20:14.631855 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:14.631822 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hv5bm_4013a04f-2b59-44fa-af92-aece71a27fc8/konnectivity-agent/0.log" limit=1024 Sep 07 16:20:14.666515 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:14.666479 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-7v9h2" event={"ID":"5d7537a7-6a94-4b85-b330-272e4cae2a70","Type":"ContainerStarted","Data":"5322f5944b464bcbc3a4e4f0adc1d4556b1b8eb3ba177a14170341f2d8a13c5b"} Sep 07 16:20:14.682752 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:14.682690 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-7v9h2" podStartSLOduration=2.202974974 podStartE2EDuration="4.6826662s" totalImagesPullingTime="2.479691226s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:20:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:20:14.682214364 +0000 UTC m=+43.717437437" watchObservedRunningTime="2026-09-07 16:20:14.6826662 +0000 UTC m=+43.717889275" Sep 07 16:20:15.820224 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:15.820177 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/89ae6799-1f6e-422c-82b3-ab64bf3967b4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kwckr\" (UID: \"89ae6799-1f6e-422c-82b3-ab64bf3967b4\") " pod="openshift-insights/insights-runtime-extractor-kwckr" Sep 07 16:20:15.820720 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:15.820239 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/89ae6799-1f6e-422c-82b3-ab64bf3967b4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kwckr\" (UID: \"89ae6799-1f6e-422c-82b3-ab64bf3967b4\") " pod="openshift-insights/insights-runtime-extractor-kwckr" Sep 07 16:20:15.820720 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:20:15.820445 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 07 16:20:15.820720 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:20:15.820520 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89ae6799-1f6e-422c-82b3-ab64bf3967b4-insights-runtime-extractor-tls podName:89ae6799-1f6e-422c-82b3-ab64bf3967b4 nodeName:}" failed. No retries permitted until 2026-09-07 16:20:23.820500887 +0000 UTC m=+52.855723937 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/89ae6799-1f6e-422c-82b3-ab64bf3967b4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kwckr" (UID: "89ae6799-1f6e-422c-82b3-ab64bf3967b4") : secret "insights-runtime-extractor-tls" not found Sep 07 16:20:15.820890 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:15.820807 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/89ae6799-1f6e-422c-82b3-ab64bf3967b4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kwckr\" (UID: \"89ae6799-1f6e-422c-82b3-ab64bf3967b4\") " pod="openshift-insights/insights-runtime-extractor-kwckr" Sep 07 16:20:15.865601 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:15.865561 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hv5bm_4013a04f-2b59-44fa-af92-aece71a27fc8/konnectivity-agent/0.log" limit=1024 Sep 07 16:20:15.996322 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:15.996293 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hv5bm_4013a04f-2b59-44fa-af92-aece71a27fc8/konnectivity-agent/0.log" limit=1024 Sep 07 16:20:16.108264 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:16.108180 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hv5bm_4013a04f-2b59-44fa-af92-aece71a27fc8/konnectivity-agent/0.log" limit=1024 Sep 07 16:20:16.197266 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:16.197232 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hv5bm_4013a04f-2b59-44fa-af92-aece71a27fc8/konnectivity-agent/0.log" limit=1024 Sep 07 16:20:16.298151 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:16.298114 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hv5bm_4013a04f-2b59-44fa-af92-aece71a27fc8/konnectivity-agent/0.log" limit=1024 Sep 07 16:20:16.448435 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:16.448404 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hv5bm_4013a04f-2b59-44fa-af92-aece71a27fc8/konnectivity-agent/0.log" limit=1024 Sep 07 16:20:16.600078 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:16.600012 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hv5bm_4013a04f-2b59-44fa-af92-aece71a27fc8/konnectivity-agent/0.log" limit=1024 Sep 07 16:20:21.164214 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:21.164170 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9-registry-tls\") pod \"image-registry-f5568c6f7-f5sdc\" (UID: \"ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9\") " pod="openshift-image-registry/image-registry-f5568c6f7-f5sdc" Sep 07 16:20:21.166816 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:21.166791 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9-registry-tls\") pod \"image-registry-f5568c6f7-f5sdc\" (UID: \"ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9\") " pod="openshift-image-registry/image-registry-f5568c6f7-f5sdc" Sep 07 16:20:21.251965 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:21.251925 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f5568c6f7-f5sdc" Sep 07 16:20:21.265077 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:21.265055 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5bc83c7d-9a35-4c79-82a4-bdad68c35006-metrics-tls\") pod \"dns-default-l2gc8\" (UID: \"5bc83c7d-9a35-4c79-82a4-bdad68c35006\") " pod="openshift-dns/dns-default-l2gc8" Sep 07 16:20:21.267570 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:21.267547 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5bc83c7d-9a35-4c79-82a4-bdad68c35006-metrics-tls\") pod \"dns-default-l2gc8\" (UID: \"5bc83c7d-9a35-4c79-82a4-bdad68c35006\") " pod="openshift-dns/dns-default-l2gc8" Sep 07 16:20:21.274499 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:21.274466 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-l2gc8" Sep 07 16:20:21.531564 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:21.531534 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-f5568c6f7-f5sdc"] Sep 07 16:20:21.534960 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:20:21.534927 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecb25d1a_0d7b_4bb0_9ee3_a80637960fa9.slice/crio-8841fa955599747e7025d2caa573263545113b5d5ef7226c614d2c2813ef5c13 WatchSource:0}: Error finding container 8841fa955599747e7025d2caa573263545113b5d5ef7226c614d2c2813ef5c13: Status 404 returned error can't find the container with id 8841fa955599747e7025d2caa573263545113b5d5ef7226c614d2c2813ef5c13 Sep 07 16:20:21.557687 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:21.557662 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-l2gc8"] Sep 07 16:20:21.560786 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:20:21.560762 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bc83c7d_9a35_4c79_82a4_bdad68c35006.slice/crio-e22a3f1b8a7da62148b1198e1c2ed234b37092d03c08a653f462180d938a32b4 WatchSource:0}: Error finding container e22a3f1b8a7da62148b1198e1c2ed234b37092d03c08a653f462180d938a32b4: Status 404 returned error can't find the container with id e22a3f1b8a7da62148b1198e1c2ed234b37092d03c08a653f462180d938a32b4 Sep 07 16:20:21.687081 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:21.687045 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f5568c6f7-f5sdc" event={"ID":"ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9","Type":"ContainerStarted","Data":"cafa312fae9fa72a3eb995b24667d206d177b1493a3afb4ed6ec6750bdcb1f4b"} Sep 07 16:20:21.687254 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:21.687105 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f5568c6f7-f5sdc" event={"ID":"ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9","Type":"ContainerStarted","Data":"8841fa955599747e7025d2caa573263545113b5d5ef7226c614d2c2813ef5c13"} Sep 07 16:20:21.687254 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:21.687130 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-f5568c6f7-f5sdc" Sep 07 16:20:21.688143 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:21.688116 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-l2gc8" event={"ID":"5bc83c7d-9a35-4c79-82a4-bdad68c35006","Type":"ContainerStarted","Data":"e22a3f1b8a7da62148b1198e1c2ed234b37092d03c08a653f462180d938a32b4"} Sep 07 16:20:21.705088 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:21.705013 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-f5568c6f7-f5sdc" podStartSLOduration=57.704998735 podStartE2EDuration="57.704998735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:19:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:20:21.704539646 +0000 UTC m=+50.739762741" watchObservedRunningTime="2026-09-07 16:20:21.704998735 +0000 UTC m=+50.740221805" Sep 07 16:20:23.696200 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:23.696170 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-l2gc8" event={"ID":"5bc83c7d-9a35-4c79-82a4-bdad68c35006","Type":"ContainerStarted","Data":"2d5b7fca52a879c3bf8b6e02d3ff9adfd61a1bbb1372b4da1e6aee4673830453"} Sep 07 16:20:23.696658 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:23.696219 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-l2gc8" event={"ID":"5bc83c7d-9a35-4c79-82a4-bdad68c35006","Type":"ContainerStarted","Data":"4e9cf0670ca0668710fba9d692a67580f021acf54c353c1372ca61b07cf88211"} Sep 07 16:20:23.696658 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:23.696312 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-l2gc8" Sep 07 16:20:23.711405 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:23.711352 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-l2gc8" podStartSLOduration=16.928756782 podStartE2EDuration="18.711338001s" totalImagesPullingTime="1.782581219s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:20:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:20:23.710814248 +0000 UTC m=+52.746037318" watchObservedRunningTime="2026-09-07 16:20:23.711338001 +0000 UTC m=+52.746561130" Sep 07 16:20:23.885702 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:23.885658 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/89ae6799-1f6e-422c-82b3-ab64bf3967b4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kwckr\" (UID: \"89ae6799-1f6e-422c-82b3-ab64bf3967b4\") " pod="openshift-insights/insights-runtime-extractor-kwckr" Sep 07 16:20:23.888152 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:23.888132 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/89ae6799-1f6e-422c-82b3-ab64bf3967b4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kwckr\" (UID: \"89ae6799-1f6e-422c-82b3-ab64bf3967b4\") " pod="openshift-insights/insights-runtime-extractor-kwckr" Sep 07 16:20:24.176975 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:24.176890 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kwckr" Sep 07 16:20:24.448347 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:24.448320 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kwckr"] Sep 07 16:20:24.452081 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:20:24.452057 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89ae6799_1f6e_422c_82b3_ab64bf3967b4.slice/crio-0c895dea3797fdc1f8e285e316f810a39fc6a22b264cd108cade3f8d38098c8f WatchSource:0}: Error finding container 0c895dea3797fdc1f8e285e316f810a39fc6a22b264cd108cade3f8d38098c8f: Status 404 returned error can't find the container with id 0c895dea3797fdc1f8e285e316f810a39fc6a22b264cd108cade3f8d38098c8f Sep 07 16:20:24.702453 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:24.702351 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kwckr" event={"ID":"89ae6799-1f6e-422c-82b3-ab64bf3967b4","Type":"ContainerStarted","Data":"6811addb5b8cbf93bafb58da8088011549e06f4b215e0681a1a13c24f09a46db"} Sep 07 16:20:24.702453 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:24.702413 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kwckr" event={"ID":"89ae6799-1f6e-422c-82b3-ab64bf3967b4","Type":"ContainerStarted","Data":"0c895dea3797fdc1f8e285e316f810a39fc6a22b264cd108cade3f8d38098c8f"} Sep 07 16:20:25.708097 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:25.708057 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kwckr" event={"ID":"89ae6799-1f6e-422c-82b3-ab64bf3967b4","Type":"ContainerStarted","Data":"d473020a1aa75e940b35ac1f9f5b5c8b14dcea281c237b3dd70d1960011fb887"} Sep 07 16:20:26.716200 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:26.716103 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kwckr" event={"ID":"89ae6799-1f6e-422c-82b3-ab64bf3967b4","Type":"ContainerStarted","Data":"946dc12ad692b09d49896cde92ead7cfec009825a6a7eb2adb75c47232b0e6f4"} Sep 07 16:20:26.739891 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:26.739842 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-kwckr" podStartSLOduration=17.846047023 podStartE2EDuration="19.73982566s" totalImagesPullingTime="1.893778637s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 16:20:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:20:26.739364416 +0000 UTC m=+55.774587488" watchObservedRunningTime="2026-09-07 16:20:26.73982566 +0000 UTC m=+55.775048731" Sep 07 16:20:33.384910 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:33.384879 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rz4mc" Sep 07 16:20:33.705169 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:33.705144 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-l2gc8" Sep 07 16:20:34.240876 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.240844 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5858cb9b95-2kvgz"] Sep 07 16:20:34.243679 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.243662 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5858cb9b95-2kvgz" Sep 07 16:20:34.245167 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.245143 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8b45b4fdc-bk4t8"] Sep 07 16:20:34.246057 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.246014 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Sep 07 16:20:34.246169 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.246082 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 07 16:20:34.246778 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.246759 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 07 16:20:34.246778 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.246778 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 07 16:20:34.247062 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.247046 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8b45b4fdc-bk4t8" Sep 07 16:20:34.250115 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.250095 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Sep 07 16:20:34.250205 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.250145 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Sep 07 16:20:34.250399 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.250381 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Sep 07 16:20:34.250939 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.250924 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Sep 07 16:20:34.261383 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.261360 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5858cb9b95-2kvgz"] Sep 07 16:20:34.262268 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.262249 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8b45b4fdc-bk4t8"] Sep 07 16:20:34.285311 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.285283 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-f5568c6f7-f5sdc"] Sep 07 16:20:34.357349 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.357307 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmf8h\" (UniqueName: \"kubernetes.io/projected/13042696-823d-4274-8f53-c1676d13d843-kube-api-access-bmf8h\") pod \"klusterlet-addon-workmgr-5858cb9b95-2kvgz\" (UID: \"13042696-823d-4274-8f53-c1676d13d843\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5858cb9b95-2kvgz" Sep 07 16:20:34.357349 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.357351 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/84cc5541-ff72-4a38-8ded-0307f4d948f7-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-8b45b4fdc-bk4t8\" (UID: \"84cc5541-ff72-4a38-8ded-0307f4d948f7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8b45b4fdc-bk4t8" Sep 07 16:20:34.357589 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.357378 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8spwf\" (UniqueName: \"kubernetes.io/projected/84cc5541-ff72-4a38-8ded-0307f4d948f7-kube-api-access-8spwf\") pod \"cluster-proxy-proxy-agent-8b45b4fdc-bk4t8\" (UID: \"84cc5541-ff72-4a38-8ded-0307f4d948f7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8b45b4fdc-bk4t8" Sep 07 16:20:34.357589 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.357447 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/84cc5541-ff72-4a38-8ded-0307f4d948f7-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-8b45b4fdc-bk4t8\" (UID: \"84cc5541-ff72-4a38-8ded-0307f4d948f7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8b45b4fdc-bk4t8" Sep 07 16:20:34.357589 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.357477 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/84cc5541-ff72-4a38-8ded-0307f4d948f7-additional-service-ca\") pod \"cluster-proxy-proxy-agent-8b45b4fdc-bk4t8\" (UID: \"84cc5541-ff72-4a38-8ded-0307f4d948f7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8b45b4fdc-bk4t8" Sep 07 16:20:34.357589 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.357518 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/13042696-823d-4274-8f53-c1676d13d843-klusterlet-config\") pod \"klusterlet-addon-workmgr-5858cb9b95-2kvgz\" (UID: \"13042696-823d-4274-8f53-c1676d13d843\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5858cb9b95-2kvgz" Sep 07 16:20:34.357589 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.357550 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/84cc5541-ff72-4a38-8ded-0307f4d948f7-ca\") pod \"cluster-proxy-proxy-agent-8b45b4fdc-bk4t8\" (UID: \"84cc5541-ff72-4a38-8ded-0307f4d948f7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8b45b4fdc-bk4t8" Sep 07 16:20:34.357801 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.357598 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13042696-823d-4274-8f53-c1676d13d843-tmp\") pod \"klusterlet-addon-workmgr-5858cb9b95-2kvgz\" (UID: \"13042696-823d-4274-8f53-c1676d13d843\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5858cb9b95-2kvgz" Sep 07 16:20:34.357801 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.357674 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/84cc5541-ff72-4a38-8ded-0307f4d948f7-hub\") pod \"cluster-proxy-proxy-agent-8b45b4fdc-bk4t8\" (UID: \"84cc5541-ff72-4a38-8ded-0307f4d948f7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8b45b4fdc-bk4t8" Sep 07 16:20:34.448179 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.448142 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-8d749d949-6k8qs"] Sep 07 16:20:34.450467 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.450441 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8d749d949-6k8qs" Sep 07 16:20:34.458579 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.458556 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/84cc5541-ff72-4a38-8ded-0307f4d948f7-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-8b45b4fdc-bk4t8\" (UID: \"84cc5541-ff72-4a38-8ded-0307f4d948f7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8b45b4fdc-bk4t8" Sep 07 16:20:34.459173 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.459146 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/84cc5541-ff72-4a38-8ded-0307f4d948f7-additional-service-ca\") pod \"cluster-proxy-proxy-agent-8b45b4fdc-bk4t8\" (UID: \"84cc5541-ff72-4a38-8ded-0307f4d948f7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8b45b4fdc-bk4t8" Sep 07 16:20:34.459251 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.459208 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/13042696-823d-4274-8f53-c1676d13d843-klusterlet-config\") pod \"klusterlet-addon-workmgr-5858cb9b95-2kvgz\" (UID: \"13042696-823d-4274-8f53-c1676d13d843\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5858cb9b95-2kvgz" Sep 07 16:20:34.459301 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.459253 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/84cc5541-ff72-4a38-8ded-0307f4d948f7-ca\") pod \"cluster-proxy-proxy-agent-8b45b4fdc-bk4t8\" (UID: \"84cc5541-ff72-4a38-8ded-0307f4d948f7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8b45b4fdc-bk4t8" Sep 07 16:20:34.459349 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.459320 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13042696-823d-4274-8f53-c1676d13d843-tmp\") pod \"klusterlet-addon-workmgr-5858cb9b95-2kvgz\" (UID: \"13042696-823d-4274-8f53-c1676d13d843\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5858cb9b95-2kvgz" Sep 07 16:20:34.459766 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.459696 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13042696-823d-4274-8f53-c1676d13d843-tmp\") pod \"klusterlet-addon-workmgr-5858cb9b95-2kvgz\" (UID: \"13042696-823d-4274-8f53-c1676d13d843\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5858cb9b95-2kvgz" Sep 07 16:20:34.459967 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.459869 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/84cc5541-ff72-4a38-8ded-0307f4d948f7-hub\") pod \"cluster-proxy-proxy-agent-8b45b4fdc-bk4t8\" (UID: \"84cc5541-ff72-4a38-8ded-0307f4d948f7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8b45b4fdc-bk4t8" Sep 07 16:20:34.459967 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.459944 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmf8h\" (UniqueName: \"kubernetes.io/projected/13042696-823d-4274-8f53-c1676d13d843-kube-api-access-bmf8h\") pod \"klusterlet-addon-workmgr-5858cb9b95-2kvgz\" (UID: \"13042696-823d-4274-8f53-c1676d13d843\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5858cb9b95-2kvgz" Sep 07 16:20:34.460120 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.459994 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/84cc5541-ff72-4a38-8ded-0307f4d948f7-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-8b45b4fdc-bk4t8\" (UID: \"84cc5541-ff72-4a38-8ded-0307f4d948f7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8b45b4fdc-bk4t8" Sep 07 16:20:34.460216 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.460170 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8spwf\" (UniqueName: \"kubernetes.io/projected/84cc5541-ff72-4a38-8ded-0307f4d948f7-kube-api-access-8spwf\") pod \"cluster-proxy-proxy-agent-8b45b4fdc-bk4t8\" (UID: \"84cc5541-ff72-4a38-8ded-0307f4d948f7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8b45b4fdc-bk4t8" Sep 07 16:20:34.460904 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.460878 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/84cc5541-ff72-4a38-8ded-0307f4d948f7-additional-service-ca\") pod \"cluster-proxy-proxy-agent-8b45b4fdc-bk4t8\" (UID: \"84cc5541-ff72-4a38-8ded-0307f4d948f7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8b45b4fdc-bk4t8" Sep 07 16:20:34.465503 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.462662 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/13042696-823d-4274-8f53-c1676d13d843-klusterlet-config\") pod \"klusterlet-addon-workmgr-5858cb9b95-2kvgz\" (UID: \"13042696-823d-4274-8f53-c1676d13d843\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5858cb9b95-2kvgz" Sep 07 16:20:34.465503 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.462698 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/84cc5541-ff72-4a38-8ded-0307f4d948f7-ca\") pod \"cluster-proxy-proxy-agent-8b45b4fdc-bk4t8\" (UID: \"84cc5541-ff72-4a38-8ded-0307f4d948f7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8b45b4fdc-bk4t8" Sep 07 16:20:34.465503 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.462701 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/84cc5541-ff72-4a38-8ded-0307f4d948f7-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-8b45b4fdc-bk4t8\" (UID: \"84cc5541-ff72-4a38-8ded-0307f4d948f7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8b45b4fdc-bk4t8" Sep 07 16:20:34.465503 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.465290 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/84cc5541-ff72-4a38-8ded-0307f4d948f7-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-8b45b4fdc-bk4t8\" (UID: \"84cc5541-ff72-4a38-8ded-0307f4d948f7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8b45b4fdc-bk4t8" Sep 07 16:20:34.467515 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.467491 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8d749d949-6k8qs"] Sep 07 16:20:34.467717 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.467700 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/84cc5541-ff72-4a38-8ded-0307f4d948f7-hub\") pod \"cluster-proxy-proxy-agent-8b45b4fdc-bk4t8\" (UID: \"84cc5541-ff72-4a38-8ded-0307f4d948f7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8b45b4fdc-bk4t8" Sep 07 16:20:34.480165 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.480142 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8spwf\" (UniqueName: \"kubernetes.io/projected/84cc5541-ff72-4a38-8ded-0307f4d948f7-kube-api-access-8spwf\") pod \"cluster-proxy-proxy-agent-8b45b4fdc-bk4t8\" (UID: \"84cc5541-ff72-4a38-8ded-0307f4d948f7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8b45b4fdc-bk4t8" Sep 07 16:20:34.480349 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.480331 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmf8h\" (UniqueName: \"kubernetes.io/projected/13042696-823d-4274-8f53-c1676d13d843-kube-api-access-bmf8h\") pod \"klusterlet-addon-workmgr-5858cb9b95-2kvgz\" (UID: \"13042696-823d-4274-8f53-c1676d13d843\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5858cb9b95-2kvgz" Sep 07 16:20:34.554885 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.554789 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5858cb9b95-2kvgz" Sep 07 16:20:34.560581 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.560559 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/945c7b16-2a9a-450e-8068-2016fb348282-image-registry-private-configuration\") pod \"image-registry-8d749d949-6k8qs\" (UID: \"945c7b16-2a9a-450e-8068-2016fb348282\") " pod="openshift-image-registry/image-registry-8d749d949-6k8qs" Sep 07 16:20:34.560637 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.560591 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/945c7b16-2a9a-450e-8068-2016fb348282-registry-tls\") pod \"image-registry-8d749d949-6k8qs\" (UID: \"945c7b16-2a9a-450e-8068-2016fb348282\") " pod="openshift-image-registry/image-registry-8d749d949-6k8qs" Sep 07 16:20:34.560637 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.560608 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/945c7b16-2a9a-450e-8068-2016fb348282-bound-sa-token\") pod \"image-registry-8d749d949-6k8qs\" (UID: \"945c7b16-2a9a-450e-8068-2016fb348282\") " pod="openshift-image-registry/image-registry-8d749d949-6k8qs" Sep 07 16:20:34.560719 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.560636 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lmh2\" (UniqueName: \"kubernetes.io/projected/945c7b16-2a9a-450e-8068-2016fb348282-kube-api-access-5lmh2\") pod \"image-registry-8d749d949-6k8qs\" (UID: \"945c7b16-2a9a-450e-8068-2016fb348282\") " pod="openshift-image-registry/image-registry-8d749d949-6k8qs" Sep 07 16:20:34.560719 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.560690 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/945c7b16-2a9a-450e-8068-2016fb348282-registry-certificates\") pod \"image-registry-8d749d949-6k8qs\" (UID: \"945c7b16-2a9a-450e-8068-2016fb348282\") " pod="openshift-image-registry/image-registry-8d749d949-6k8qs" Sep 07 16:20:34.560719 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.560714 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/945c7b16-2a9a-450e-8068-2016fb348282-trusted-ca\") pod \"image-registry-8d749d949-6k8qs\" (UID: \"945c7b16-2a9a-450e-8068-2016fb348282\") " pod="openshift-image-registry/image-registry-8d749d949-6k8qs" Sep 07 16:20:34.560831 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.560774 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/945c7b16-2a9a-450e-8068-2016fb348282-ca-trust-extracted\") pod \"image-registry-8d749d949-6k8qs\" (UID: \"945c7b16-2a9a-450e-8068-2016fb348282\") " pod="openshift-image-registry/image-registry-8d749d949-6k8qs" Sep 07 16:20:34.560866 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.560852 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/945c7b16-2a9a-450e-8068-2016fb348282-installation-pull-secrets\") pod \"image-registry-8d749d949-6k8qs\" (UID: \"945c7b16-2a9a-450e-8068-2016fb348282\") " pod="openshift-image-registry/image-registry-8d749d949-6k8qs" Sep 07 16:20:34.567571 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.567543 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8b45b4fdc-bk4t8" Sep 07 16:20:34.661573 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.661540 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/945c7b16-2a9a-450e-8068-2016fb348282-ca-trust-extracted\") pod \"image-registry-8d749d949-6k8qs\" (UID: \"945c7b16-2a9a-450e-8068-2016fb348282\") " pod="openshift-image-registry/image-registry-8d749d949-6k8qs" Sep 07 16:20:34.661728 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.661596 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/945c7b16-2a9a-450e-8068-2016fb348282-installation-pull-secrets\") pod \"image-registry-8d749d949-6k8qs\" (UID: \"945c7b16-2a9a-450e-8068-2016fb348282\") " pod="openshift-image-registry/image-registry-8d749d949-6k8qs" Sep 07 16:20:34.661728 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.661667 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/945c7b16-2a9a-450e-8068-2016fb348282-image-registry-private-configuration\") pod \"image-registry-8d749d949-6k8qs\" (UID: \"945c7b16-2a9a-450e-8068-2016fb348282\") " pod="openshift-image-registry/image-registry-8d749d949-6k8qs" Sep 07 16:20:34.661728 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.661694 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/945c7b16-2a9a-450e-8068-2016fb348282-registry-tls\") pod \"image-registry-8d749d949-6k8qs\" (UID: \"945c7b16-2a9a-450e-8068-2016fb348282\") " pod="openshift-image-registry/image-registry-8d749d949-6k8qs" Sep 07 16:20:34.661728 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.661716 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/945c7b16-2a9a-450e-8068-2016fb348282-bound-sa-token\") pod \"image-registry-8d749d949-6k8qs\" (UID: \"945c7b16-2a9a-450e-8068-2016fb348282\") " pod="openshift-image-registry/image-registry-8d749d949-6k8qs" Sep 07 16:20:34.661858 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.661744 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lmh2\" (UniqueName: \"kubernetes.io/projected/945c7b16-2a9a-450e-8068-2016fb348282-kube-api-access-5lmh2\") pod \"image-registry-8d749d949-6k8qs\" (UID: \"945c7b16-2a9a-450e-8068-2016fb348282\") " pod="openshift-image-registry/image-registry-8d749d949-6k8qs" Sep 07 16:20:34.661858 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.661786 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/945c7b16-2a9a-450e-8068-2016fb348282-registry-certificates\") pod \"image-registry-8d749d949-6k8qs\" (UID: \"945c7b16-2a9a-450e-8068-2016fb348282\") " pod="openshift-image-registry/image-registry-8d749d949-6k8qs" Sep 07 16:20:34.661858 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.661818 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/945c7b16-2a9a-450e-8068-2016fb348282-trusted-ca\") pod \"image-registry-8d749d949-6k8qs\" (UID: \"945c7b16-2a9a-450e-8068-2016fb348282\") " pod="openshift-image-registry/image-registry-8d749d949-6k8qs" Sep 07 16:20:34.662134 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.661997 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/945c7b16-2a9a-450e-8068-2016fb348282-ca-trust-extracted\") pod \"image-registry-8d749d949-6k8qs\" (UID: \"945c7b16-2a9a-450e-8068-2016fb348282\") " pod="openshift-image-registry/image-registry-8d749d949-6k8qs" Sep 07 16:20:34.662728 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.662702 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/945c7b16-2a9a-450e-8068-2016fb348282-registry-certificates\") pod \"image-registry-8d749d949-6k8qs\" (UID: \"945c7b16-2a9a-450e-8068-2016fb348282\") " pod="openshift-image-registry/image-registry-8d749d949-6k8qs" Sep 07 16:20:34.662962 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.662938 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/945c7b16-2a9a-450e-8068-2016fb348282-trusted-ca\") pod \"image-registry-8d749d949-6k8qs\" (UID: \"945c7b16-2a9a-450e-8068-2016fb348282\") " pod="openshift-image-registry/image-registry-8d749d949-6k8qs" Sep 07 16:20:34.664359 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.664339 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/945c7b16-2a9a-450e-8068-2016fb348282-installation-pull-secrets\") pod \"image-registry-8d749d949-6k8qs\" (UID: \"945c7b16-2a9a-450e-8068-2016fb348282\") " pod="openshift-image-registry/image-registry-8d749d949-6k8qs" Sep 07 16:20:34.664825 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.664804 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/945c7b16-2a9a-450e-8068-2016fb348282-registry-tls\") pod \"image-registry-8d749d949-6k8qs\" (UID: \"945c7b16-2a9a-450e-8068-2016fb348282\") " pod="openshift-image-registry/image-registry-8d749d949-6k8qs" Sep 07 16:20:34.664906 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.664854 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/945c7b16-2a9a-450e-8068-2016fb348282-image-registry-private-configuration\") pod \"image-registry-8d749d949-6k8qs\" (UID: \"945c7b16-2a9a-450e-8068-2016fb348282\") " pod="openshift-image-registry/image-registry-8d749d949-6k8qs" Sep 07 16:20:34.676039 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.675988 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lmh2\" (UniqueName: \"kubernetes.io/projected/945c7b16-2a9a-450e-8068-2016fb348282-kube-api-access-5lmh2\") pod \"image-registry-8d749d949-6k8qs\" (UID: \"945c7b16-2a9a-450e-8068-2016fb348282\") " pod="openshift-image-registry/image-registry-8d749d949-6k8qs" Sep 07 16:20:34.676595 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.676572 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/945c7b16-2a9a-450e-8068-2016fb348282-bound-sa-token\") pod \"image-registry-8d749d949-6k8qs\" (UID: \"945c7b16-2a9a-450e-8068-2016fb348282\") " pod="openshift-image-registry/image-registry-8d749d949-6k8qs" Sep 07 16:20:34.760069 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.760008 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8d749d949-6k8qs" Sep 07 16:20:34.824894 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.824823 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5858cb9b95-2kvgz"] Sep 07 16:20:34.828584 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:20:34.828557 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13042696_823d_4274_8f53_c1676d13d843.slice/crio-beea73b86933ce1b17a3d27eabae88b3495167f35304800f45baadde366acd6a WatchSource:0}: Error finding container beea73b86933ce1b17a3d27eabae88b3495167f35304800f45baadde366acd6a: Status 404 returned error can't find the container with id beea73b86933ce1b17a3d27eabae88b3495167f35304800f45baadde366acd6a Sep 07 16:20:34.852861 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:34.852772 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8b45b4fdc-bk4t8"] Sep 07 16:20:34.857274 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:20:34.857249 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod84cc5541_ff72_4a38_8ded_0307f4d948f7.slice/crio-3fa8bc102de20f0316b23c0f8ca906439d5a8498a5dd7602d086d137b12717ba WatchSource:0}: Error finding container 3fa8bc102de20f0316b23c0f8ca906439d5a8498a5dd7602d086d137b12717ba: Status 404 returned error can't find the container with id 3fa8bc102de20f0316b23c0f8ca906439d5a8498a5dd7602d086d137b12717ba Sep 07 16:20:35.028188 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:35.028150 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8d749d949-6k8qs"] Sep 07 16:20:35.031691 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:20:35.031663 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod945c7b16_2a9a_450e_8068_2016fb348282.slice/crio-3a86270364289ba5663bb107e0c0db5cab8e979d34aad0736d92873b2cf02f16 WatchSource:0}: Error finding container 3a86270364289ba5663bb107e0c0db5cab8e979d34aad0736d92873b2cf02f16: Status 404 returned error can't find the container with id 3a86270364289ba5663bb107e0c0db5cab8e979d34aad0736d92873b2cf02f16 Sep 07 16:20:35.743660 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:35.743620 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8b45b4fdc-bk4t8" event={"ID":"84cc5541-ff72-4a38-8ded-0307f4d948f7","Type":"ContainerStarted","Data":"3fa8bc102de20f0316b23c0f8ca906439d5a8498a5dd7602d086d137b12717ba"} Sep 07 16:20:35.745738 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:35.745647 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8d749d949-6k8qs" event={"ID":"945c7b16-2a9a-450e-8068-2016fb348282","Type":"ContainerStarted","Data":"18375191e6746cd2a13a261e08248e059190902daf42708275a6c5c149d10118"} Sep 07 16:20:35.745863 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:35.745757 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8d749d949-6k8qs" event={"ID":"945c7b16-2a9a-450e-8068-2016fb348282","Type":"ContainerStarted","Data":"3a86270364289ba5663bb107e0c0db5cab8e979d34aad0736d92873b2cf02f16"} Sep 07 16:20:35.745863 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:35.745787 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-8d749d949-6k8qs" Sep 07 16:20:35.747145 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:35.747121 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5858cb9b95-2kvgz" event={"ID":"13042696-823d-4274-8f53-c1676d13d843","Type":"ContainerStarted","Data":"beea73b86933ce1b17a3d27eabae88b3495167f35304800f45baadde366acd6a"} Sep 07 16:20:35.763484 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:35.763422 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-8d749d949-6k8qs" podStartSLOduration=1.7633987009999998 podStartE2EDuration="1.763398701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:20:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:20:35.761849115 +0000 UTC m=+64.797072200" watchObservedRunningTime="2026-09-07 16:20:35.763398701 +0000 UTC m=+64.798621775" Sep 07 16:20:37.287094 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:37.287051 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/599e026d-7338-4319-a1d7-7b714bbb6e58-metrics-certs\") pod \"network-metrics-daemon-lhv86\" (UID: \"599e026d-7338-4319-a1d7-7b714bbb6e58\") " pod="openshift-multus/network-metrics-daemon-lhv86" Sep 07 16:20:37.287518 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:37.287107 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcgl7\" (UniqueName: \"kubernetes.io/projected/552ec3ab-d017-450f-85bb-616f65146d2e-kube-api-access-kcgl7\") pod \"network-check-target-8g844\" (UID: \"552ec3ab-d017-450f-85bb-616f65146d2e\") " pod="openshift-network-diagnostics/network-check-target-8g844" Sep 07 16:20:37.287518 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:37.287243 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4f4aaec6-7ab4-48ab-a0e1-4d26977f0315-original-pull-secret\") pod \"global-pull-secret-syncer-6n92f\" (UID: \"4f4aaec6-7ab4-48ab-a0e1-4d26977f0315\") " pod="kube-system/global-pull-secret-syncer-6n92f" Sep 07 16:20:37.289307 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:37.289284 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 07 16:20:37.289491 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:37.289286 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 07 16:20:37.290083 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:37.290064 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 07 16:20:37.299313 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:37.299292 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 07 16:20:37.302633 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:37.302610 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/599e026d-7338-4319-a1d7-7b714bbb6e58-metrics-certs\") pod \"network-metrics-daemon-lhv86\" (UID: \"599e026d-7338-4319-a1d7-7b714bbb6e58\") " pod="openshift-multus/network-metrics-daemon-lhv86" Sep 07 16:20:37.303219 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:37.303201 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4f4aaec6-7ab4-48ab-a0e1-4d26977f0315-original-pull-secret\") pod \"global-pull-secret-syncer-6n92f\" (UID: \"4f4aaec6-7ab4-48ab-a0e1-4d26977f0315\") " pod="kube-system/global-pull-secret-syncer-6n92f" Sep 07 16:20:37.311895 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:37.311870 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcgl7\" (UniqueName: \"kubernetes.io/projected/552ec3ab-d017-450f-85bb-616f65146d2e-kube-api-access-kcgl7\") pod \"network-check-target-8g844\" (UID: \"552ec3ab-d017-450f-85bb-616f65146d2e\") " pod="openshift-network-diagnostics/network-check-target-8g844" Sep 07 16:20:37.528006 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:37.527969 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-m4f7n\"" Sep 07 16:20:37.534829 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:37.534797 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9lqj4\"" Sep 07 16:20:37.536097 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:37.536073 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhv86" Sep 07 16:20:37.538389 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:37.537943 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6n92f" Sep 07 16:20:37.543916 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:37.543890 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8g844" Sep 07 16:20:40.771328 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:40.771293 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8b45b4fdc-bk4t8" event={"ID":"84cc5541-ff72-4a38-8ded-0307f4d948f7","Type":"ContainerStarted","Data":"0c91c25dc4a188f69e870cf13cc63bf25567d8e495ae6f43e679d84d358c7fe2"} Sep 07 16:20:40.771678 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:40.771346 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8b45b4fdc-bk4t8" event={"ID":"84cc5541-ff72-4a38-8ded-0307f4d948f7","Type":"ContainerStarted","Data":"c9331ea3847a49f0b59cbd968fc12acd9e34184a4048825bb53fbdb19ac461ad"} Sep 07 16:20:40.772872 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:40.772835 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5858cb9b95-2kvgz" event={"ID":"13042696-823d-4274-8f53-c1676d13d843","Type":"ContainerStarted","Data":"6ffcf9186808a97ec994a78cacaae9b654260c3900347596cd8fc3e438edc706"} Sep 07 16:20:40.773016 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:40.773001 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5858cb9b95-2kvgz" Sep 07 16:20:40.774866 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:40.774846 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5858cb9b95-2kvgz" Sep 07 16:20:40.786617 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:40.786572 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5858cb9b95-2kvgz" podStartSLOduration=1.048679765 podStartE2EDuration="6.786553444s" totalImagesPullingTime="5.737873679s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:20:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:20:40.786263691 +0000 UTC m=+69.821486777" watchObservedRunningTime="2026-09-07 16:20:40.786553444 +0000 UTC m=+69.821776516" Sep 07 16:20:40.830322 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:40.830271 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6n92f"] Sep 07 16:20:40.834057 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:20:40.834015 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f4aaec6_7ab4_48ab_a0e1_4d26977f0315.slice/crio-7cc6c2ff3874372311c9ea27539e6e4c21c59ce92fea6c524ee3a90702a03f50 WatchSource:0}: Error finding container 7cc6c2ff3874372311c9ea27539e6e4c21c59ce92fea6c524ee3a90702a03f50: Status 404 returned error can't find the container with id 7cc6c2ff3874372311c9ea27539e6e4c21c59ce92fea6c524ee3a90702a03f50 Sep 07 16:20:40.853777 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:40.853753 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lhv86"] Sep 07 16:20:40.856276 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:20:40.856254 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod599e026d_7338_4319_a1d7_7b714bbb6e58.slice/crio-eaa6a5ff9a9236af954556909021bf48b76098c215bf1d58ca73872bbedde9d9 WatchSource:0}: Error finding container eaa6a5ff9a9236af954556909021bf48b76098c215bf1d58ca73872bbedde9d9: Status 404 returned error can't find the container with id eaa6a5ff9a9236af954556909021bf48b76098c215bf1d58ca73872bbedde9d9 Sep 07 16:20:40.865058 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:40.865034 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8g844"] Sep 07 16:20:40.867743 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:20:40.867717 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod552ec3ab_d017_450f_85bb_616f65146d2e.slice/crio-7862be9506b8a6341d9d6ea3ac1af5a800df42a183ae60f51581a875f109ad08 WatchSource:0}: Error finding container 7862be9506b8a6341d9d6ea3ac1af5a800df42a183ae60f51581a875f109ad08: Status 404 returned error can't find the container with id 7862be9506b8a6341d9d6ea3ac1af5a800df42a183ae60f51581a875f109ad08 Sep 07 16:20:41.778758 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:41.778716 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6n92f" event={"ID":"4f4aaec6-7ab4-48ab-a0e1-4d26977f0315","Type":"ContainerStarted","Data":"7cc6c2ff3874372311c9ea27539e6e4c21c59ce92fea6c524ee3a90702a03f50"} Sep 07 16:20:41.780212 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:41.780177 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8g844" event={"ID":"552ec3ab-d017-450f-85bb-616f65146d2e","Type":"ContainerStarted","Data":"7862be9506b8a6341d9d6ea3ac1af5a800df42a183ae60f51581a875f109ad08"} Sep 07 16:20:41.781741 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:41.781710 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lhv86" event={"ID":"599e026d-7338-4319-a1d7-7b714bbb6e58","Type":"ContainerStarted","Data":"eaa6a5ff9a9236af954556909021bf48b76098c215bf1d58ca73872bbedde9d9"} Sep 07 16:20:41.785977 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:41.785763 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8b45b4fdc-bk4t8" event={"ID":"84cc5541-ff72-4a38-8ded-0307f4d948f7","Type":"ContainerStarted","Data":"74113ec0dfe23ef5bc7a3defb88244e700b0ae0be63990479ed3622c33ab3bd8"} Sep 07 16:20:41.804440 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:41.804370 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8b45b4fdc-bk4t8" podStartSLOduration=2.093655215 podStartE2EDuration="7.804348892s" totalImagesPullingTime="5.710693677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:20:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:20:41.802305095 +0000 UTC m=+70.837528167" watchObservedRunningTime="2026-09-07 16:20:41.804348892 +0000 UTC m=+70.839571963" Sep 07 16:20:42.793496 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:42.793456 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lhv86" event={"ID":"599e026d-7338-4319-a1d7-7b714bbb6e58","Type":"ContainerStarted","Data":"db7dde5c4b9ed2d31232960e480da47891e33a21e76fd60c52f482ae87a70c64"} Sep 07 16:20:42.793990 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:42.793554 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lhv86" event={"ID":"599e026d-7338-4319-a1d7-7b714bbb6e58","Type":"ContainerStarted","Data":"ecdba6ca892ecac9c8ae36fb39deaa830c5c3ee661029cbc62967c8d66d0c6f4"} Sep 07 16:20:42.809733 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:42.809670 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-lhv86" podStartSLOduration=70.69666841 podStartE2EDuration="1m11.80965085s" totalImagesPullingTime="1.11298244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:19:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:20:42.808960503 +0000 UTC m=+71.844183575" watchObservedRunningTime="2026-09-07 16:20:42.80965085 +0000 UTC m=+71.844873920" Sep 07 16:20:44.292439 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:44.292406 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-f5568c6f7-f5sdc" Sep 07 16:20:46.811269 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:46.811223 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8g844" event={"ID":"552ec3ab-d017-450f-85bb-616f65146d2e","Type":"ContainerStarted","Data":"51084f51affb356d435016b18b2f5fe60adef2ea3c018c958483fca8a350810b"} Sep 07 16:20:46.811714 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:46.811347 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8g844" Sep 07 16:20:46.812797 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:46.812772 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6n92f" event={"ID":"4f4aaec6-7ab4-48ab-a0e1-4d26977f0315","Type":"ContainerStarted","Data":"e52ec9bccde9f5c7ab358692573066b367885d92d1919c0752155003232c4f35"} Sep 07 16:20:46.825208 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:46.825160 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8g844" podStartSLOduration=70.915009493 podStartE2EDuration="1m15.825142934s" totalImagesPullingTime="4.910133441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:19:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:20:46.824097385 +0000 UTC m=+75.859320457" watchObservedRunningTime="2026-09-07 16:20:46.825142934 +0000 UTC m=+75.860366006" Sep 07 16:20:46.837340 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:46.837294 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-6n92f" podStartSLOduration=70.890541526 podStartE2EDuration="1m15.837277061s" totalImagesPullingTime="4.946735535s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 16:19:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:20:46.835931455 +0000 UTC m=+75.871154527" watchObservedRunningTime="2026-09-07 16:20:46.837277061 +0000 UTC m=+75.872500134" Sep 07 16:20:47.355675 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:47.355633 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-cpc9m"] Sep 07 16:20:47.358032 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:47.358003 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cpc9m" Sep 07 16:20:47.360079 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:47.360058 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 07 16:20:47.360343 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:47.360324 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2glkw\"" Sep 07 16:20:47.360429 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:47.360391 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 07 16:20:47.360481 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:47.360443 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 07 16:20:47.361164 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:47.361119 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 07 16:20:47.361262 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:47.361183 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 07 16:20:47.361262 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:47.361219 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 07 16:20:47.482904 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:47.482870 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/04187790-e6b1-4739-94d5-a7d20645ae1c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cpc9m\" (UID: \"04187790-e6b1-4739-94d5-a7d20645ae1c\") " pod="openshift-monitoring/node-exporter-cpc9m" Sep 07 16:20:47.482904 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:47.482907 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/04187790-e6b1-4739-94d5-a7d20645ae1c-node-exporter-accelerators-collector-config\") pod \"node-exporter-cpc9m\" (UID: \"04187790-e6b1-4739-94d5-a7d20645ae1c\") " pod="openshift-monitoring/node-exporter-cpc9m" Sep 07 16:20:47.483170 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:47.482927 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/04187790-e6b1-4739-94d5-a7d20645ae1c-sys\") pod \"node-exporter-cpc9m\" (UID: \"04187790-e6b1-4739-94d5-a7d20645ae1c\") " pod="openshift-monitoring/node-exporter-cpc9m" Sep 07 16:20:47.483170 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:47.482994 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/04187790-e6b1-4739-94d5-a7d20645ae1c-node-exporter-wtmp\") pod \"node-exporter-cpc9m\" (UID: \"04187790-e6b1-4739-94d5-a7d20645ae1c\") " pod="openshift-monitoring/node-exporter-cpc9m" Sep 07 16:20:47.483170 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:47.483054 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/04187790-e6b1-4739-94d5-a7d20645ae1c-root\") pod \"node-exporter-cpc9m\" (UID: \"04187790-e6b1-4739-94d5-a7d20645ae1c\") " pod="openshift-monitoring/node-exporter-cpc9m" Sep 07 16:20:47.483170 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:47.483098 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/04187790-e6b1-4739-94d5-a7d20645ae1c-metrics-client-ca\") pod \"node-exporter-cpc9m\" (UID: \"04187790-e6b1-4739-94d5-a7d20645ae1c\") " pod="openshift-monitoring/node-exporter-cpc9m" Sep 07 16:20:47.483300 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:47.483174 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/04187790-e6b1-4739-94d5-a7d20645ae1c-node-exporter-textfile\") pod \"node-exporter-cpc9m\" (UID: \"04187790-e6b1-4739-94d5-a7d20645ae1c\") " pod="openshift-monitoring/node-exporter-cpc9m" Sep 07 16:20:47.483300 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:47.483203 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/04187790-e6b1-4739-94d5-a7d20645ae1c-node-exporter-tls\") pod \"node-exporter-cpc9m\" (UID: \"04187790-e6b1-4739-94d5-a7d20645ae1c\") " pod="openshift-monitoring/node-exporter-cpc9m" Sep 07 16:20:47.483300 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:47.483234 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb9p2\" (UniqueName: \"kubernetes.io/projected/04187790-e6b1-4739-94d5-a7d20645ae1c-kube-api-access-tb9p2\") pod \"node-exporter-cpc9m\" (UID: \"04187790-e6b1-4739-94d5-a7d20645ae1c\") " pod="openshift-monitoring/node-exporter-cpc9m" Sep 07 16:20:47.584283 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:47.584244 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb9p2\" (UniqueName: \"kubernetes.io/projected/04187790-e6b1-4739-94d5-a7d20645ae1c-kube-api-access-tb9p2\") pod \"node-exporter-cpc9m\" (UID: \"04187790-e6b1-4739-94d5-a7d20645ae1c\") " pod="openshift-monitoring/node-exporter-cpc9m" Sep 07 16:20:47.584460 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:47.584296 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/04187790-e6b1-4739-94d5-a7d20645ae1c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cpc9m\" (UID: \"04187790-e6b1-4739-94d5-a7d20645ae1c\") " pod="openshift-monitoring/node-exporter-cpc9m" Sep 07 16:20:47.584460 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:47.584326 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/04187790-e6b1-4739-94d5-a7d20645ae1c-node-exporter-accelerators-collector-config\") pod \"node-exporter-cpc9m\" (UID: \"04187790-e6b1-4739-94d5-a7d20645ae1c\") " pod="openshift-monitoring/node-exporter-cpc9m" Sep 07 16:20:47.584460 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:47.584355 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/04187790-e6b1-4739-94d5-a7d20645ae1c-sys\") pod \"node-exporter-cpc9m\" (UID: \"04187790-e6b1-4739-94d5-a7d20645ae1c\") " pod="openshift-monitoring/node-exporter-cpc9m" Sep 07 16:20:47.584460 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:47.584384 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/04187790-e6b1-4739-94d5-a7d20645ae1c-node-exporter-wtmp\") pod \"node-exporter-cpc9m\" (UID: \"04187790-e6b1-4739-94d5-a7d20645ae1c\") " pod="openshift-monitoring/node-exporter-cpc9m" Sep 07 16:20:47.584460 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:47.584412 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/04187790-e6b1-4739-94d5-a7d20645ae1c-root\") pod \"node-exporter-cpc9m\" (UID: \"04187790-e6b1-4739-94d5-a7d20645ae1c\") " pod="openshift-monitoring/node-exporter-cpc9m" Sep 07 16:20:47.584460 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:47.584446 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/04187790-e6b1-4739-94d5-a7d20645ae1c-metrics-client-ca\") pod \"node-exporter-cpc9m\" (UID: \"04187790-e6b1-4739-94d5-a7d20645ae1c\") " pod="openshift-monitoring/node-exporter-cpc9m" Sep 07 16:20:47.584828 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:47.584461 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/04187790-e6b1-4739-94d5-a7d20645ae1c-sys\") pod \"node-exporter-cpc9m\" (UID: \"04187790-e6b1-4739-94d5-a7d20645ae1c\") " pod="openshift-monitoring/node-exporter-cpc9m" Sep 07 16:20:47.584828 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:47.584487 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/04187790-e6b1-4739-94d5-a7d20645ae1c-node-exporter-textfile\") pod \"node-exporter-cpc9m\" (UID: \"04187790-e6b1-4739-94d5-a7d20645ae1c\") " pod="openshift-monitoring/node-exporter-cpc9m" Sep 07 16:20:47.584828 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:47.584515 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/04187790-e6b1-4739-94d5-a7d20645ae1c-node-exporter-tls\") pod \"node-exporter-cpc9m\" (UID: \"04187790-e6b1-4739-94d5-a7d20645ae1c\") " pod="openshift-monitoring/node-exporter-cpc9m" Sep 07 16:20:47.584828 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:47.584523 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/04187790-e6b1-4739-94d5-a7d20645ae1c-root\") pod \"node-exporter-cpc9m\" (UID: \"04187790-e6b1-4739-94d5-a7d20645ae1c\") " pod="openshift-monitoring/node-exporter-cpc9m" Sep 07 16:20:47.584828 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:47.584528 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/04187790-e6b1-4739-94d5-a7d20645ae1c-node-exporter-wtmp\") pod \"node-exporter-cpc9m\" (UID: \"04187790-e6b1-4739-94d5-a7d20645ae1c\") " pod="openshift-monitoring/node-exporter-cpc9m" Sep 07 16:20:47.585105 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:47.584837 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/04187790-e6b1-4739-94d5-a7d20645ae1c-node-exporter-textfile\") pod \"node-exporter-cpc9m\" (UID: \"04187790-e6b1-4739-94d5-a7d20645ae1c\") " pod="openshift-monitoring/node-exporter-cpc9m" Sep 07 16:20:47.585105 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:47.585011 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/04187790-e6b1-4739-94d5-a7d20645ae1c-node-exporter-accelerators-collector-config\") pod \"node-exporter-cpc9m\" (UID: \"04187790-e6b1-4739-94d5-a7d20645ae1c\") " pod="openshift-monitoring/node-exporter-cpc9m" Sep 07 16:20:47.585105 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:47.585065 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/04187790-e6b1-4739-94d5-a7d20645ae1c-metrics-client-ca\") pod \"node-exporter-cpc9m\" (UID: \"04187790-e6b1-4739-94d5-a7d20645ae1c\") " pod="openshift-monitoring/node-exporter-cpc9m" Sep 07 16:20:47.587062 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:47.587014 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/04187790-e6b1-4739-94d5-a7d20645ae1c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cpc9m\" (UID: \"04187790-e6b1-4739-94d5-a7d20645ae1c\") " pod="openshift-monitoring/node-exporter-cpc9m" Sep 07 16:20:47.587203 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:47.587185 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/04187790-e6b1-4739-94d5-a7d20645ae1c-node-exporter-tls\") pod \"node-exporter-cpc9m\" (UID: \"04187790-e6b1-4739-94d5-a7d20645ae1c\") " pod="openshift-monitoring/node-exporter-cpc9m" Sep 07 16:20:47.595937 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:47.595915 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb9p2\" (UniqueName: \"kubernetes.io/projected/04187790-e6b1-4739-94d5-a7d20645ae1c-kube-api-access-tb9p2\") pod \"node-exporter-cpc9m\" (UID: \"04187790-e6b1-4739-94d5-a7d20645ae1c\") " pod="openshift-monitoring/node-exporter-cpc9m" Sep 07 16:20:47.669312 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:47.669221 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cpc9m" Sep 07 16:20:47.682054 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:20:47.681999 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04187790_e6b1_4739_94d5_a7d20645ae1c.slice/crio-c06d121f6bb77762c340f32bd9b700cfa0883777e7d2115eb7712b3bd81ad9e2 WatchSource:0}: Error finding container c06d121f6bb77762c340f32bd9b700cfa0883777e7d2115eb7712b3bd81ad9e2: Status 404 returned error can't find the container with id c06d121f6bb77762c340f32bd9b700cfa0883777e7d2115eb7712b3bd81ad9e2 Sep 07 16:20:47.818506 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:47.818465 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cpc9m" event={"ID":"04187790-e6b1-4739-94d5-a7d20645ae1c","Type":"ContainerStarted","Data":"c06d121f6bb77762c340f32bd9b700cfa0883777e7d2115eb7712b3bd81ad9e2"} Sep 07 16:20:49.827856 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:49.827824 2580 generic.go:356] "Generic (PLEG): container finished" podID="04187790-e6b1-4739-94d5-a7d20645ae1c" containerID="3b4190c1c11e1c481e6abf5b16276474878a34c8f8aa1fe6b45b0124f2757b64" exitCode=0 Sep 07 16:20:49.828273 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:49.827870 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cpc9m" event={"ID":"04187790-e6b1-4739-94d5-a7d20645ae1c","Type":"ContainerDied","Data":"3b4190c1c11e1c481e6abf5b16276474878a34c8f8aa1fe6b45b0124f2757b64"} Sep 07 16:20:50.833785 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:50.833748 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cpc9m" event={"ID":"04187790-e6b1-4739-94d5-a7d20645ae1c","Type":"ContainerStarted","Data":"df6d137b35f1c275bd9ce7ea51598b25d19176108034b74437bdafaff0fe86a0"} Sep 07 16:20:50.834229 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:50.833826 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cpc9m" event={"ID":"04187790-e6b1-4739-94d5-a7d20645ae1c","Type":"ContainerStarted","Data":"2d980c4317999c8e9672ef0fd622b1f8c82f8cebe79abef5a487ca305dd9092e"} Sep 07 16:20:50.851900 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:50.851841 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-cpc9m" podStartSLOduration=2.6854864960000002 podStartE2EDuration="3.851824951s" totalImagesPullingTime="1.094912025s" totalInitContainerRuntime="71.42643ms" isStatefulPod=true podCreationTimestamp="2026-09-07 16:20:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:20:50.850443511 +0000 UTC m=+79.885666576" watchObservedRunningTime="2026-09-07 16:20:50.851824951 +0000 UTC m=+79.887048022" Sep 07 16:20:56.757358 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:56.757252 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-8d749d949-6k8qs" Sep 07 16:20:59.304498 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:59.304457 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-f5568c6f7-f5sdc" podUID="ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9" containerName="registry" containerID="cri-o://cafa312fae9fa72a3eb995b24667d206d177b1493a3afb4ed6ec6750bdcb1f4b" gracePeriod=30 Sep 07 16:20:59.555290 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:59.555205 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f5568c6f7-f5sdc" Sep 07 16:20:59.681864 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:59.681818 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9-trusted-ca\") pod \"ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9\" (UID: \"ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9\") " Sep 07 16:20:59.682075 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:59.681885 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9-ca-trust-extracted\") pod \"ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9\" (UID: \"ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9\") " Sep 07 16:20:59.682075 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:59.681908 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9-image-registry-private-configuration\") pod \"ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9\" (UID: \"ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9\") " Sep 07 16:20:59.682075 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:59.681942 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9-installation-pull-secrets\") pod \"ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9\" (UID: \"ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9\") " Sep 07 16:20:59.682075 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:59.681966 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9-registry-certificates\") pod \"ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9\" (UID: \"ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9\") " Sep 07 16:20:59.682075 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:59.682000 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9-bound-sa-token\") pod \"ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9\" (UID: \"ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9\") " Sep 07 16:20:59.682075 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:59.682046 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9-registry-tls\") pod \"ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9\" (UID: \"ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9\") " Sep 07 16:20:59.682378 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:59.682100 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wlj5w\" (UniqueName: \"kubernetes.io/projected/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9-kube-api-access-wlj5w\") pod \"ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9\" (UID: \"ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9\") " Sep 07 16:20:59.682527 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:59.682466 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9" (UID: "ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 16:20:59.682641 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:59.682531 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9" (UID: "ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 16:20:59.684813 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:59.684760 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9-kube-api-access-wlj5w" (OuterVolumeSpecName: "kube-api-access-wlj5w") pod "ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9" (UID: "ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9"). InnerVolumeSpecName "kube-api-access-wlj5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 16:20:59.684924 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:59.684883 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9" (UID: "ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 16:20:59.685000 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:59.684940 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9" (UID: "ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 16:20:59.685000 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:59.684955 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9" (UID: "ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 16:20:59.685204 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:59.685182 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9" (UID: "ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 16:20:59.691355 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:59.691326 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9" (UID: "ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 16:20:59.782887 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:59.782847 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9-trusted-ca\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:20:59.782887 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:59.782880 2580 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9-ca-trust-extracted\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:20:59.782887 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:59.782892 2580 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9-image-registry-private-configuration\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:20:59.783152 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:59.782901 2580 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9-installation-pull-secrets\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:20:59.783152 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:59.782911 2580 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9-registry-certificates\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:20:59.783152 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:59.782919 2580 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9-bound-sa-token\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:20:59.783152 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:59.782927 2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9-registry-tls\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:20:59.783152 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:59.782935 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlj5w\" (UniqueName: \"kubernetes.io/projected/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9-kube-api-access-wlj5w\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:20:59.866546 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:59.866453 2580 generic.go:356] "Generic (PLEG): container finished" podID="ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9" containerID="cafa312fae9fa72a3eb995b24667d206d177b1493a3afb4ed6ec6750bdcb1f4b" exitCode=0 Sep 07 16:20:59.866546 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:59.866502 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f5568c6f7-f5sdc" Sep 07 16:20:59.866546 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:59.866531 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f5568c6f7-f5sdc" event={"ID":"ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9","Type":"ContainerDied","Data":"cafa312fae9fa72a3eb995b24667d206d177b1493a3afb4ed6ec6750bdcb1f4b"} Sep 07 16:20:59.866767 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:59.866610 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f5568c6f7-f5sdc" event={"ID":"ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9","Type":"ContainerDied","Data":"8841fa955599747e7025d2caa573263545113b5d5ef7226c614d2c2813ef5c13"} Sep 07 16:20:59.866767 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:59.866636 2580 scope.go:117] "RemoveContainer" containerID="cafa312fae9fa72a3eb995b24667d206d177b1493a3afb4ed6ec6750bdcb1f4b" Sep 07 16:20:59.875977 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:59.875957 2580 scope.go:117] "RemoveContainer" containerID="cafa312fae9fa72a3eb995b24667d206d177b1493a3afb4ed6ec6750bdcb1f4b" Sep 07 16:20:59.876322 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:20:59.876303 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cafa312fae9fa72a3eb995b24667d206d177b1493a3afb4ed6ec6750bdcb1f4b\": container with ID starting with cafa312fae9fa72a3eb995b24667d206d177b1493a3afb4ed6ec6750bdcb1f4b not found: ID does not exist" containerID="cafa312fae9fa72a3eb995b24667d206d177b1493a3afb4ed6ec6750bdcb1f4b" Sep 07 16:20:59.876373 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:59.876331 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cafa312fae9fa72a3eb995b24667d206d177b1493a3afb4ed6ec6750bdcb1f4b"} err="failed to get container status \"cafa312fae9fa72a3eb995b24667d206d177b1493a3afb4ed6ec6750bdcb1f4b\": rpc error: code = NotFound desc = could not find container \"cafa312fae9fa72a3eb995b24667d206d177b1493a3afb4ed6ec6750bdcb1f4b\": container with ID starting with cafa312fae9fa72a3eb995b24667d206d177b1493a3afb4ed6ec6750bdcb1f4b not found: ID does not exist" Sep 07 16:20:59.885789 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:59.885756 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-f5568c6f7-f5sdc"] Sep 07 16:20:59.889184 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:20:59.889155 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-f5568c6f7-f5sdc"] Sep 07 16:21:01.417429 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:21:01.417390 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9" path="/var/lib/kubelet/pods/ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9/volumes" Sep 07 16:21:17.821364 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:21:17.821333 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8g844" Sep 07 16:23:20.752724 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:20.752692 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ppn5t"] Sep 07 16:23:20.753189 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:20.752957 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9" containerName="registry" Sep 07 16:23:20.753189 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:20.752968 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9" containerName="registry" Sep 07 16:23:20.753189 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:20.753009 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecb25d1a-0d7b-4bb0-9ee3-a80637960fa9" containerName="registry" Sep 07 16:23:20.754659 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:20.754643 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ppn5t" Sep 07 16:23:20.756893 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:20.756874 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 07 16:23:20.757504 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:20.757490 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-hzscn\"" Sep 07 16:23:20.757588 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:20.757531 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 07 16:23:20.762118 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:20.762092 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ppn5t"] Sep 07 16:23:20.785146 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:20.785101 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpnrt\" (UniqueName: \"kubernetes.io/projected/8759d9b5-5657-4082-8318-6340fd432a25-kube-api-access-fpnrt\") pod \"openshift-lws-operator-fd8ccff4c-ppn5t\" (UID: \"8759d9b5-5657-4082-8318-6340fd432a25\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ppn5t" Sep 07 16:23:20.785321 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:20.785162 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8759d9b5-5657-4082-8318-6340fd432a25-tmp\") pod \"openshift-lws-operator-fd8ccff4c-ppn5t\" (UID: \"8759d9b5-5657-4082-8318-6340fd432a25\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ppn5t" Sep 07 16:23:20.886118 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:20.886086 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpnrt\" (UniqueName: \"kubernetes.io/projected/8759d9b5-5657-4082-8318-6340fd432a25-kube-api-access-fpnrt\") pod \"openshift-lws-operator-fd8ccff4c-ppn5t\" (UID: \"8759d9b5-5657-4082-8318-6340fd432a25\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ppn5t" Sep 07 16:23:20.886284 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:20.886136 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8759d9b5-5657-4082-8318-6340fd432a25-tmp\") pod \"openshift-lws-operator-fd8ccff4c-ppn5t\" (UID: \"8759d9b5-5657-4082-8318-6340fd432a25\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ppn5t" Sep 07 16:23:20.886475 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:20.886460 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8759d9b5-5657-4082-8318-6340fd432a25-tmp\") pod \"openshift-lws-operator-fd8ccff4c-ppn5t\" (UID: \"8759d9b5-5657-4082-8318-6340fd432a25\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ppn5t" Sep 07 16:23:20.894070 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:20.894038 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpnrt\" (UniqueName: \"kubernetes.io/projected/8759d9b5-5657-4082-8318-6340fd432a25-kube-api-access-fpnrt\") pod \"openshift-lws-operator-fd8ccff4c-ppn5t\" (UID: \"8759d9b5-5657-4082-8318-6340fd432a25\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ppn5t" Sep 07 16:23:21.064480 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:21.064397 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ppn5t" Sep 07 16:23:21.347506 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:21.347430 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ppn5t"] Sep 07 16:23:21.350903 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:23:21.350874 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8759d9b5_5657_4082_8318_6340fd432a25.slice/crio-a775f7df0f32e62f68578f5ceb8b2fb19cc33b47d22cd3c34e350dbfb14033bb WatchSource:0}: Error finding container a775f7df0f32e62f68578f5ceb8b2fb19cc33b47d22cd3c34e350dbfb14033bb: Status 404 returned error can't find the container with id a775f7df0f32e62f68578f5ceb8b2fb19cc33b47d22cd3c34e350dbfb14033bb Sep 07 16:23:22.355546 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:22.355503 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ppn5t" event={"ID":"8759d9b5-5657-4082-8318-6340fd432a25","Type":"ContainerStarted","Data":"a775f7df0f32e62f68578f5ceb8b2fb19cc33b47d22cd3c34e350dbfb14033bb"} Sep 07 16:23:24.364556 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:24.364461 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ppn5t" event={"ID":"8759d9b5-5657-4082-8318-6340fd432a25","Type":"ContainerStarted","Data":"57b1f72c75bf6fcad3529194aa09bef895328108df9b396f5911e85e4bc9790b"} Sep 07 16:23:24.379253 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:24.379203 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ppn5t" podStartSLOduration=1.6902324 podStartE2EDuration="4.379185904s" totalImagesPullingTime="2.688953504s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:23:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:23:24.378585984 +0000 UTC m=+233.413809058" watchObservedRunningTime="2026-09-07 16:23:24.379185904 +0000 UTC m=+233.414408975" Sep 07 16:23:39.536114 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:39.536075 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-57d49c86b4-xp4kf"] Sep 07 16:23:39.540303 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:39.540283 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-57d49c86b4-xp4kf" Sep 07 16:23:39.542932 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:39.542905 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 07 16:23:39.543896 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:39.543875 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-q76fw\"" Sep 07 16:23:39.544054 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:39.543972 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 07 16:23:39.544054 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:39.544042 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 07 16:23:39.558786 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:39.558748 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-57d49c86b4-xp4kf"] Sep 07 16:23:39.628534 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:39.628501 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd7pl\" (UniqueName: \"kubernetes.io/projected/363da8d9-ef7a-4eee-9ef4-1f7f4fc60414-kube-api-access-xd7pl\") pod \"lws-controller-manager-57d49c86b4-xp4kf\" (UID: \"363da8d9-ef7a-4eee-9ef4-1f7f4fc60414\") " pod="openshift-lws-operator/lws-controller-manager-57d49c86b4-xp4kf" Sep 07 16:23:39.628720 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:39.628546 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/363da8d9-ef7a-4eee-9ef4-1f7f4fc60414-metrics-cert\") pod \"lws-controller-manager-57d49c86b4-xp4kf\" (UID: \"363da8d9-ef7a-4eee-9ef4-1f7f4fc60414\") " pod="openshift-lws-operator/lws-controller-manager-57d49c86b4-xp4kf" Sep 07 16:23:39.628720 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:39.628609 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/363da8d9-ef7a-4eee-9ef4-1f7f4fc60414-cert\") pod \"lws-controller-manager-57d49c86b4-xp4kf\" (UID: \"363da8d9-ef7a-4eee-9ef4-1f7f4fc60414\") " pod="openshift-lws-operator/lws-controller-manager-57d49c86b4-xp4kf" Sep 07 16:23:39.628720 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:39.628663 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/363da8d9-ef7a-4eee-9ef4-1f7f4fc60414-manager-config\") pod \"lws-controller-manager-57d49c86b4-xp4kf\" (UID: \"363da8d9-ef7a-4eee-9ef4-1f7f4fc60414\") " pod="openshift-lws-operator/lws-controller-manager-57d49c86b4-xp4kf" Sep 07 16:23:39.729154 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:39.729120 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/363da8d9-ef7a-4eee-9ef4-1f7f4fc60414-manager-config\") pod \"lws-controller-manager-57d49c86b4-xp4kf\" (UID: \"363da8d9-ef7a-4eee-9ef4-1f7f4fc60414\") " pod="openshift-lws-operator/lws-controller-manager-57d49c86b4-xp4kf" Sep 07 16:23:39.729154 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:39.729158 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd7pl\" (UniqueName: \"kubernetes.io/projected/363da8d9-ef7a-4eee-9ef4-1f7f4fc60414-kube-api-access-xd7pl\") pod \"lws-controller-manager-57d49c86b4-xp4kf\" (UID: \"363da8d9-ef7a-4eee-9ef4-1f7f4fc60414\") " pod="openshift-lws-operator/lws-controller-manager-57d49c86b4-xp4kf" Sep 07 16:23:39.729365 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:39.729188 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/363da8d9-ef7a-4eee-9ef4-1f7f4fc60414-metrics-cert\") pod \"lws-controller-manager-57d49c86b4-xp4kf\" (UID: \"363da8d9-ef7a-4eee-9ef4-1f7f4fc60414\") " pod="openshift-lws-operator/lws-controller-manager-57d49c86b4-xp4kf" Sep 07 16:23:39.729365 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:39.729233 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/363da8d9-ef7a-4eee-9ef4-1f7f4fc60414-cert\") pod \"lws-controller-manager-57d49c86b4-xp4kf\" (UID: \"363da8d9-ef7a-4eee-9ef4-1f7f4fc60414\") " pod="openshift-lws-operator/lws-controller-manager-57d49c86b4-xp4kf" Sep 07 16:23:39.729816 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:39.729793 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/363da8d9-ef7a-4eee-9ef4-1f7f4fc60414-manager-config\") pod \"lws-controller-manager-57d49c86b4-xp4kf\" (UID: \"363da8d9-ef7a-4eee-9ef4-1f7f4fc60414\") " pod="openshift-lws-operator/lws-controller-manager-57d49c86b4-xp4kf" Sep 07 16:23:39.732006 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:39.731984 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/363da8d9-ef7a-4eee-9ef4-1f7f4fc60414-metrics-cert\") pod \"lws-controller-manager-57d49c86b4-xp4kf\" (UID: \"363da8d9-ef7a-4eee-9ef4-1f7f4fc60414\") " pod="openshift-lws-operator/lws-controller-manager-57d49c86b4-xp4kf" Sep 07 16:23:39.732114 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:39.731981 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/363da8d9-ef7a-4eee-9ef4-1f7f4fc60414-cert\") pod \"lws-controller-manager-57d49c86b4-xp4kf\" (UID: \"363da8d9-ef7a-4eee-9ef4-1f7f4fc60414\") " pod="openshift-lws-operator/lws-controller-manager-57d49c86b4-xp4kf" Sep 07 16:23:39.738006 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:39.737985 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd7pl\" (UniqueName: \"kubernetes.io/projected/363da8d9-ef7a-4eee-9ef4-1f7f4fc60414-kube-api-access-xd7pl\") pod \"lws-controller-manager-57d49c86b4-xp4kf\" (UID: \"363da8d9-ef7a-4eee-9ef4-1f7f4fc60414\") " pod="openshift-lws-operator/lws-controller-manager-57d49c86b4-xp4kf" Sep 07 16:23:39.849586 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:39.849500 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-57d49c86b4-xp4kf" Sep 07 16:23:40.120557 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:40.120372 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-57d49c86b4-xp4kf"] Sep 07 16:23:40.123203 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:23:40.123177 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod363da8d9_ef7a_4eee_9ef4_1f7f4fc60414.slice/crio-d5563a51ccf902514841e1f31ea4036532171f58651aaa37435249aaeb3a8960 WatchSource:0}: Error finding container d5563a51ccf902514841e1f31ea4036532171f58651aaa37435249aaeb3a8960: Status 404 returned error can't find the container with id d5563a51ccf902514841e1f31ea4036532171f58651aaa37435249aaeb3a8960 Sep 07 16:23:40.421383 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:40.421298 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-57d49c86b4-xp4kf" event={"ID":"363da8d9-ef7a-4eee-9ef4-1f7f4fc60414","Type":"ContainerStarted","Data":"d5563a51ccf902514841e1f31ea4036532171f58651aaa37435249aaeb3a8960"} Sep 07 16:23:42.430501 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:42.430468 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-57d49c86b4-xp4kf" event={"ID":"363da8d9-ef7a-4eee-9ef4-1f7f4fc60414","Type":"ContainerStarted","Data":"9a888ed7aa8e1392e74ccb544c05416780162c787e337a70a7343be29bf1910b"} Sep 07 16:23:42.430932 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:42.430547 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-57d49c86b4-xp4kf" Sep 07 16:23:42.444387 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:42.444334 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-57d49c86b4-xp4kf" podStartSLOduration=1.318700281 podStartE2EDuration="3.444315144s" totalImagesPullingTime="2.125614863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:23:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:23:42.443694979 +0000 UTC m=+251.478918051" watchObservedRunningTime="2026-09-07 16:23:42.444315144 +0000 UTC m=+251.479538213" Sep 07 16:23:44.518106 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:44.518064 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7766cc5489-hcs8n"] Sep 07 16:23:44.520252 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:44.520230 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7766cc5489-hcs8n" Sep 07 16:23:44.523700 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:44.523681 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 07 16:23:44.523821 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:44.523805 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 07 16:23:44.523894 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:44.523806 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 07 16:23:44.523894 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:44.523857 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-f6lx2\"" Sep 07 16:23:44.524158 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:44.524142 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 07 16:23:44.544002 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:44.543974 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7766cc5489-hcs8n"] Sep 07 16:23:44.564787 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:44.564754 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c454a42-b87a-4061-b104-522b821e3978-tmp\") pod \"opendatahub-operator-controller-manager-7766cc5489-hcs8n\" (UID: \"8c454a42-b87a-4061-b104-522b821e3978\") " pod="opendatahub/opendatahub-operator-controller-manager-7766cc5489-hcs8n" Sep 07 16:23:44.564957 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:44.564803 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8c454a42-b87a-4061-b104-522b821e3978-webhook-cert\") pod \"opendatahub-operator-controller-manager-7766cc5489-hcs8n\" (UID: \"8c454a42-b87a-4061-b104-522b821e3978\") " pod="opendatahub/opendatahub-operator-controller-manager-7766cc5489-hcs8n" Sep 07 16:23:44.564957 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:44.564858 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8c454a42-b87a-4061-b104-522b821e3978-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7766cc5489-hcs8n\" (UID: \"8c454a42-b87a-4061-b104-522b821e3978\") " pod="opendatahub/opendatahub-operator-controller-manager-7766cc5489-hcs8n" Sep 07 16:23:44.564957 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:44.564887 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp7sx\" (UniqueName: \"kubernetes.io/projected/8c454a42-b87a-4061-b104-522b821e3978-kube-api-access-bp7sx\") pod \"opendatahub-operator-controller-manager-7766cc5489-hcs8n\" (UID: \"8c454a42-b87a-4061-b104-522b821e3978\") " pod="opendatahub/opendatahub-operator-controller-manager-7766cc5489-hcs8n" Sep 07 16:23:44.564957 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:44.564941 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8c454a42-b87a-4061-b104-522b821e3978-manifests\") pod \"opendatahub-operator-controller-manager-7766cc5489-hcs8n\" (UID: \"8c454a42-b87a-4061-b104-522b821e3978\") " pod="opendatahub/opendatahub-operator-controller-manager-7766cc5489-hcs8n" Sep 07 16:23:44.665937 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:44.665877 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8c454a42-b87a-4061-b104-522b821e3978-webhook-cert\") pod \"opendatahub-operator-controller-manager-7766cc5489-hcs8n\" (UID: \"8c454a42-b87a-4061-b104-522b821e3978\") " pod="opendatahub/opendatahub-operator-controller-manager-7766cc5489-hcs8n" Sep 07 16:23:44.665937 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:44.665946 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8c454a42-b87a-4061-b104-522b821e3978-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7766cc5489-hcs8n\" (UID: \"8c454a42-b87a-4061-b104-522b821e3978\") " pod="opendatahub/opendatahub-operator-controller-manager-7766cc5489-hcs8n" Sep 07 16:23:44.666198 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:44.665968 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp7sx\" (UniqueName: \"kubernetes.io/projected/8c454a42-b87a-4061-b104-522b821e3978-kube-api-access-bp7sx\") pod \"opendatahub-operator-controller-manager-7766cc5489-hcs8n\" (UID: \"8c454a42-b87a-4061-b104-522b821e3978\") " pod="opendatahub/opendatahub-operator-controller-manager-7766cc5489-hcs8n" Sep 07 16:23:44.666198 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:44.665988 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8c454a42-b87a-4061-b104-522b821e3978-manifests\") pod \"opendatahub-operator-controller-manager-7766cc5489-hcs8n\" (UID: \"8c454a42-b87a-4061-b104-522b821e3978\") " pod="opendatahub/opendatahub-operator-controller-manager-7766cc5489-hcs8n" Sep 07 16:23:44.666198 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:44.666043 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c454a42-b87a-4061-b104-522b821e3978-tmp\") pod \"opendatahub-operator-controller-manager-7766cc5489-hcs8n\" (UID: \"8c454a42-b87a-4061-b104-522b821e3978\") " pod="opendatahub/opendatahub-operator-controller-manager-7766cc5489-hcs8n" Sep 07 16:23:44.666531 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:44.666507 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8c454a42-b87a-4061-b104-522b821e3978-manifests\") pod \"opendatahub-operator-controller-manager-7766cc5489-hcs8n\" (UID: \"8c454a42-b87a-4061-b104-522b821e3978\") " pod="opendatahub/opendatahub-operator-controller-manager-7766cc5489-hcs8n" Sep 07 16:23:44.668503 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:44.668474 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c454a42-b87a-4061-b104-522b821e3978-tmp\") pod \"opendatahub-operator-controller-manager-7766cc5489-hcs8n\" (UID: \"8c454a42-b87a-4061-b104-522b821e3978\") " pod="opendatahub/opendatahub-operator-controller-manager-7766cc5489-hcs8n" Sep 07 16:23:44.668632 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:44.668594 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8c454a42-b87a-4061-b104-522b821e3978-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7766cc5489-hcs8n\" (UID: \"8c454a42-b87a-4061-b104-522b821e3978\") " pod="opendatahub/opendatahub-operator-controller-manager-7766cc5489-hcs8n" Sep 07 16:23:44.668759 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:44.668739 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8c454a42-b87a-4061-b104-522b821e3978-webhook-cert\") pod \"opendatahub-operator-controller-manager-7766cc5489-hcs8n\" (UID: \"8c454a42-b87a-4061-b104-522b821e3978\") " pod="opendatahub/opendatahub-operator-controller-manager-7766cc5489-hcs8n" Sep 07 16:23:44.673445 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:44.673425 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp7sx\" (UniqueName: \"kubernetes.io/projected/8c454a42-b87a-4061-b104-522b821e3978-kube-api-access-bp7sx\") pod \"opendatahub-operator-controller-manager-7766cc5489-hcs8n\" (UID: \"8c454a42-b87a-4061-b104-522b821e3978\") " pod="opendatahub/opendatahub-operator-controller-manager-7766cc5489-hcs8n" Sep 07 16:23:44.830626 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:44.830539 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7766cc5489-hcs8n" Sep 07 16:23:45.112180 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:45.112101 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7766cc5489-hcs8n"] Sep 07 16:23:45.115899 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:23:45.115867 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c454a42_b87a_4061_b104_522b821e3978.slice/crio-798f172bbe1ecf883773490ea1cd0cb3c7ee99e9d7f93ca044e90c828a048a77 WatchSource:0}: Error finding container 798f172bbe1ecf883773490ea1cd0cb3c7ee99e9d7f93ca044e90c828a048a77: Status 404 returned error can't find the container with id 798f172bbe1ecf883773490ea1cd0cb3c7ee99e9d7f93ca044e90c828a048a77 Sep 07 16:23:45.442013 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:45.441927 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7766cc5489-hcs8n" event={"ID":"8c454a42-b87a-4061-b104-522b821e3978","Type":"ContainerStarted","Data":"798f172bbe1ecf883773490ea1cd0cb3c7ee99e9d7f93ca044e90c828a048a77"} Sep 07 16:23:48.455076 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:48.455037 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7766cc5489-hcs8n" event={"ID":"8c454a42-b87a-4061-b104-522b821e3978","Type":"ContainerStarted","Data":"754f831cdbf28120395e85e90ef2158ac0e2ef55e9d14515f747aaed2c2ed228"} Sep 07 16:23:49.460137 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:49.460103 2580 generic.go:356] "Generic (PLEG): container finished" podID="8c454a42-b87a-4061-b104-522b821e3978" containerID="754f831cdbf28120395e85e90ef2158ac0e2ef55e9d14515f747aaed2c2ed228" exitCode=0 Sep 07 16:23:49.460499 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:49.460187 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7766cc5489-hcs8n" event={"ID":"8c454a42-b87a-4061-b104-522b821e3978","Type":"ContainerDied","Data":"754f831cdbf28120395e85e90ef2158ac0e2ef55e9d14515f747aaed2c2ed228"} Sep 07 16:23:50.465559 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:50.465531 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7766cc5489-hcs8n" event={"ID":"8c454a42-b87a-4061-b104-522b821e3978","Type":"ContainerStarted","Data":"83d6bfc988a8791b4e34cdb6420dd85f43ccf615dedab9ad4a9984a288d0a970"} Sep 07 16:23:50.465931 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:50.465655 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7766cc5489-hcs8n" Sep 07 16:23:50.484242 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:50.484195 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7766cc5489-hcs8n" podStartSLOduration=1.895997078 podStartE2EDuration="6.484178007s" totalImagesPullingTime="3.586087508s" totalInitContainerRuntime="1.002093421s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:23:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:23:50.483182011 +0000 UTC m=+259.518405093" watchObservedRunningTime="2026-09-07 16:23:50.484178007 +0000 UTC m=+259.519401076" Sep 07 16:23:53.435536 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:23:53.435463 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-57d49c86b4-xp4kf" Sep 07 16:24:01.471685 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:01.471658 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7766cc5489-hcs8n" Sep 07 16:24:11.009387 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:11.009353 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg"] Sep 07 16:24:11.016391 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:11.016369 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg" Sep 07 16:24:11.018428 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:11.018400 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 07 16:24:11.018584 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:11.018567 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-zx6ch\"" Sep 07 16:24:11.018648 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:11.018587 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 07 16:24:11.018698 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:11.018646 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 07 16:24:11.026715 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:11.026694 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg"] Sep 07 16:24:11.172405 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:11.172368 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fe982b2-5675-4a4b-b8a7-5cc0c397480c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg\" (UID: \"8fe982b2-5675-4a4b-b8a7-5cc0c397480c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg" Sep 07 16:24:11.172405 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:11.172403 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fe982b2-5675-4a4b-b8a7-5cc0c397480c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg\" (UID: \"8fe982b2-5675-4a4b-b8a7-5cc0c397480c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg" Sep 07 16:24:11.172718 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:11.172442 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fe982b2-5675-4a4b-b8a7-5cc0c397480c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg\" (UID: \"8fe982b2-5675-4a4b-b8a7-5cc0c397480c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg" Sep 07 16:24:11.172718 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:11.172497 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fe982b2-5675-4a4b-b8a7-5cc0c397480c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg\" (UID: \"8fe982b2-5675-4a4b-b8a7-5cc0c397480c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg" Sep 07 16:24:11.172718 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:11.172572 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fe982b2-5675-4a4b-b8a7-5cc0c397480c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg\" (UID: \"8fe982b2-5675-4a4b-b8a7-5cc0c397480c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg" Sep 07 16:24:11.172718 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:11.172592 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fe982b2-5675-4a4b-b8a7-5cc0c397480c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg\" (UID: \"8fe982b2-5675-4a4b-b8a7-5cc0c397480c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg" Sep 07 16:24:11.172718 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:11.172608 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fe982b2-5675-4a4b-b8a7-5cc0c397480c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg\" (UID: \"8fe982b2-5675-4a4b-b8a7-5cc0c397480c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg" Sep 07 16:24:11.172718 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:11.172622 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fe982b2-5675-4a4b-b8a7-5cc0c397480c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg\" (UID: \"8fe982b2-5675-4a4b-b8a7-5cc0c397480c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg" Sep 07 16:24:11.172718 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:11.172637 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg9ft\" (UniqueName: \"kubernetes.io/projected/8fe982b2-5675-4a4b-b8a7-5cc0c397480c-kube-api-access-cg9ft\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg\" (UID: \"8fe982b2-5675-4a4b-b8a7-5cc0c397480c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg" Sep 07 16:24:11.274154 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:11.274014 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fe982b2-5675-4a4b-b8a7-5cc0c397480c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg\" (UID: \"8fe982b2-5675-4a4b-b8a7-5cc0c397480c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg" Sep 07 16:24:11.274154 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:11.274113 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fe982b2-5675-4a4b-b8a7-5cc0c397480c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg\" (UID: \"8fe982b2-5675-4a4b-b8a7-5cc0c397480c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg" Sep 07 16:24:11.274154 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:11.274142 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fe982b2-5675-4a4b-b8a7-5cc0c397480c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg\" (UID: \"8fe982b2-5675-4a4b-b8a7-5cc0c397480c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg" Sep 07 16:24:11.274476 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:11.274168 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fe982b2-5675-4a4b-b8a7-5cc0c397480c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg\" (UID: \"8fe982b2-5675-4a4b-b8a7-5cc0c397480c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg" Sep 07 16:24:11.274476 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:11.274202 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg9ft\" (UniqueName: \"kubernetes.io/projected/8fe982b2-5675-4a4b-b8a7-5cc0c397480c-kube-api-access-cg9ft\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg\" (UID: \"8fe982b2-5675-4a4b-b8a7-5cc0c397480c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg" Sep 07 16:24:11.274476 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:11.274273 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fe982b2-5675-4a4b-b8a7-5cc0c397480c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg\" (UID: \"8fe982b2-5675-4a4b-b8a7-5cc0c397480c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg" Sep 07 16:24:11.274476 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:11.274300 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fe982b2-5675-4a4b-b8a7-5cc0c397480c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg\" (UID: \"8fe982b2-5675-4a4b-b8a7-5cc0c397480c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg" Sep 07 16:24:11.274476 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:11.274357 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fe982b2-5675-4a4b-b8a7-5cc0c397480c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg\" (UID: \"8fe982b2-5675-4a4b-b8a7-5cc0c397480c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg" Sep 07 16:24:11.274476 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:11.274383 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fe982b2-5675-4a4b-b8a7-5cc0c397480c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg\" (UID: \"8fe982b2-5675-4a4b-b8a7-5cc0c397480c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg" Sep 07 16:24:11.274780 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:11.274589 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fe982b2-5675-4a4b-b8a7-5cc0c397480c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg\" (UID: \"8fe982b2-5675-4a4b-b8a7-5cc0c397480c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg" Sep 07 16:24:11.274780 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:11.274652 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fe982b2-5675-4a4b-b8a7-5cc0c397480c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg\" (UID: \"8fe982b2-5675-4a4b-b8a7-5cc0c397480c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg" Sep 07 16:24:11.274879 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:11.274848 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fe982b2-5675-4a4b-b8a7-5cc0c397480c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg\" (UID: \"8fe982b2-5675-4a4b-b8a7-5cc0c397480c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg" Sep 07 16:24:11.274986 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:11.274927 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fe982b2-5675-4a4b-b8a7-5cc0c397480c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg\" (UID: \"8fe982b2-5675-4a4b-b8a7-5cc0c397480c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg" Sep 07 16:24:11.275252 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:11.275227 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fe982b2-5675-4a4b-b8a7-5cc0c397480c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg\" (UID: \"8fe982b2-5675-4a4b-b8a7-5cc0c397480c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg" Sep 07 16:24:11.276806 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:11.276782 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fe982b2-5675-4a4b-b8a7-5cc0c397480c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg\" (UID: \"8fe982b2-5675-4a4b-b8a7-5cc0c397480c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg" Sep 07 16:24:11.277063 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:11.277046 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fe982b2-5675-4a4b-b8a7-5cc0c397480c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg\" (UID: \"8fe982b2-5675-4a4b-b8a7-5cc0c397480c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg" Sep 07 16:24:11.281316 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:11.281293 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fe982b2-5675-4a4b-b8a7-5cc0c397480c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg\" (UID: \"8fe982b2-5675-4a4b-b8a7-5cc0c397480c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg" Sep 07 16:24:11.281673 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:11.281646 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg9ft\" (UniqueName: \"kubernetes.io/projected/8fe982b2-5675-4a4b-b8a7-5cc0c397480c-kube-api-access-cg9ft\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg\" (UID: \"8fe982b2-5675-4a4b-b8a7-5cc0c397480c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg" Sep 07 16:24:11.327953 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:11.327916 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg" Sep 07 16:24:11.605799 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:11.605723 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg"] Sep 07 16:24:11.608826 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:24:11.608796 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fe982b2_5675_4a4b_b8a7_5cc0c397480c.slice/crio-52e66c97e0df56679c465771bb6b73eea42234bb61be084b3586d23a9bd438b8 WatchSource:0}: Error finding container 52e66c97e0df56679c465771bb6b73eea42234bb61be084b3586d23a9bd438b8: Status 404 returned error can't find the container with id 52e66c97e0df56679c465771bb6b73eea42234bb61be084b3586d23a9bd438b8 Sep 07 16:24:12.549099 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:12.549048 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg" event={"ID":"8fe982b2-5675-4a4b-b8a7-5cc0c397480c","Type":"ContainerStarted","Data":"52e66c97e0df56679c465771bb6b73eea42234bb61be084b3586d23a9bd438b8"} Sep 07 16:24:14.285517 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:14.285484 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:24:14.285723 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:14.285556 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:24:14.285723 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:14.285592 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:24:14.559117 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:14.559040 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg" event={"ID":"8fe982b2-5675-4a4b-b8a7-5cc0c397480c","Type":"ContainerStarted","Data":"6c569ebc43a6bb63bb5a2ed5872a15323b4f70dfdf7b3fbef08a7c9993f5534a"} Sep 07 16:24:14.576715 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:14.576666 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg" podStartSLOduration=1.902323212 podStartE2EDuration="4.576649885s" totalImagesPullingTime="2.674326673s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:24:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:24:14.575481632 +0000 UTC m=+283.610704706" watchObservedRunningTime="2026-09-07 16:24:14.576649885 +0000 UTC m=+283.611872957" Sep 07 16:24:15.328547 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:15.328497 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg" Sep 07 16:24:15.333518 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:15.333490 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg" Sep 07 16:24:15.563381 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:15.563348 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg" Sep 07 16:24:15.564504 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:15.564482 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg" Sep 07 16:24:21.589058 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:21.589010 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-2rc48"] Sep 07 16:24:21.596096 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:21.596076 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2rc48" Sep 07 16:24:21.598706 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:21.598677 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-fc7kl\"" Sep 07 16:24:21.598923 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:21.598902 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 07 16:24:21.599000 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:21.598982 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 07 16:24:21.599180 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:21.599161 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 07 16:24:21.603769 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:21.603743 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-2rc48"] Sep 07 16:24:21.662836 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:21.662794 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnh9s\" (UniqueName: \"kubernetes.io/projected/182ffcc1-816e-4871-a357-6b9ac07e4481-kube-api-access-gnh9s\") pod \"istiod-openshift-gateway-5677ccb49b-2rc48\" (UID: \"182ffcc1-816e-4871-a357-6b9ac07e4481\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2rc48" Sep 07 16:24:21.662836 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:21.662833 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/182ffcc1-816e-4871-a357-6b9ac07e4481-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-2rc48\" (UID: \"182ffcc1-816e-4871-a357-6b9ac07e4481\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2rc48" Sep 07 16:24:21.663099 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:21.662885 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/182ffcc1-816e-4871-a357-6b9ac07e4481-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-2rc48\" (UID: \"182ffcc1-816e-4871-a357-6b9ac07e4481\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2rc48" Sep 07 16:24:21.663099 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:21.662926 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/182ffcc1-816e-4871-a357-6b9ac07e4481-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-2rc48\" (UID: \"182ffcc1-816e-4871-a357-6b9ac07e4481\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2rc48" Sep 07 16:24:21.663099 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:21.662959 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/182ffcc1-816e-4871-a357-6b9ac07e4481-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-2rc48\" (UID: \"182ffcc1-816e-4871-a357-6b9ac07e4481\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2rc48" Sep 07 16:24:21.663099 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:21.663000 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/182ffcc1-816e-4871-a357-6b9ac07e4481-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-2rc48\" (UID: \"182ffcc1-816e-4871-a357-6b9ac07e4481\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2rc48" Sep 07 16:24:21.663099 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:21.663088 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/182ffcc1-816e-4871-a357-6b9ac07e4481-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-2rc48\" (UID: \"182ffcc1-816e-4871-a357-6b9ac07e4481\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2rc48" Sep 07 16:24:21.763831 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:21.763792 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/182ffcc1-816e-4871-a357-6b9ac07e4481-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-2rc48\" (UID: \"182ffcc1-816e-4871-a357-6b9ac07e4481\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2rc48" Sep 07 16:24:21.764050 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:21.763846 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/182ffcc1-816e-4871-a357-6b9ac07e4481-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-2rc48\" (UID: \"182ffcc1-816e-4871-a357-6b9ac07e4481\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2rc48" Sep 07 16:24:21.764050 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:21.763886 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnh9s\" (UniqueName: \"kubernetes.io/projected/182ffcc1-816e-4871-a357-6b9ac07e4481-kube-api-access-gnh9s\") pod \"istiod-openshift-gateway-5677ccb49b-2rc48\" (UID: \"182ffcc1-816e-4871-a357-6b9ac07e4481\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2rc48" Sep 07 16:24:21.764050 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:21.763916 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/182ffcc1-816e-4871-a357-6b9ac07e4481-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-2rc48\" (UID: \"182ffcc1-816e-4871-a357-6b9ac07e4481\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2rc48" Sep 07 16:24:21.764050 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:21.763941 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/182ffcc1-816e-4871-a357-6b9ac07e4481-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-2rc48\" (UID: \"182ffcc1-816e-4871-a357-6b9ac07e4481\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2rc48" Sep 07 16:24:21.764050 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:21.763971 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/182ffcc1-816e-4871-a357-6b9ac07e4481-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-2rc48\" (UID: \"182ffcc1-816e-4871-a357-6b9ac07e4481\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2rc48" Sep 07 16:24:21.764050 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:21.764002 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/182ffcc1-816e-4871-a357-6b9ac07e4481-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-2rc48\" (UID: \"182ffcc1-816e-4871-a357-6b9ac07e4481\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2rc48" Sep 07 16:24:21.764627 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:21.764596 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/182ffcc1-816e-4871-a357-6b9ac07e4481-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-2rc48\" (UID: \"182ffcc1-816e-4871-a357-6b9ac07e4481\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2rc48" Sep 07 16:24:21.766552 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:21.766524 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/182ffcc1-816e-4871-a357-6b9ac07e4481-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-2rc48\" (UID: \"182ffcc1-816e-4871-a357-6b9ac07e4481\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2rc48" Sep 07 16:24:21.766668 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:21.766579 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/182ffcc1-816e-4871-a357-6b9ac07e4481-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-2rc48\" (UID: \"182ffcc1-816e-4871-a357-6b9ac07e4481\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2rc48" Sep 07 16:24:21.766668 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:21.766584 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/182ffcc1-816e-4871-a357-6b9ac07e4481-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-2rc48\" (UID: \"182ffcc1-816e-4871-a357-6b9ac07e4481\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2rc48" Sep 07 16:24:21.766773 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:21.766697 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/182ffcc1-816e-4871-a357-6b9ac07e4481-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-2rc48\" (UID: \"182ffcc1-816e-4871-a357-6b9ac07e4481\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2rc48" Sep 07 16:24:21.771968 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:21.771942 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnh9s\" (UniqueName: \"kubernetes.io/projected/182ffcc1-816e-4871-a357-6b9ac07e4481-kube-api-access-gnh9s\") pod \"istiod-openshift-gateway-5677ccb49b-2rc48\" (UID: \"182ffcc1-816e-4871-a357-6b9ac07e4481\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2rc48" Sep 07 16:24:21.772361 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:21.772343 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/182ffcc1-816e-4871-a357-6b9ac07e4481-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-2rc48\" (UID: \"182ffcc1-816e-4871-a357-6b9ac07e4481\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2rc48" Sep 07 16:24:21.906080 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:21.905955 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2rc48" Sep 07 16:24:22.184575 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:22.184500 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-2rc48"] Sep 07 16:24:22.187647 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:24:22.187620 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod182ffcc1_816e_4871_a357_6b9ac07e4481.slice/crio-407af4968868a1e049dc67a13340908d132b34879a8950c923b38449c409ea3d WatchSource:0}: Error finding container 407af4968868a1e049dc67a13340908d132b34879a8950c923b38449c409ea3d: Status 404 returned error can't find the container with id 407af4968868a1e049dc67a13340908d132b34879a8950c923b38449c409ea3d Sep 07 16:24:22.595940 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:22.595901 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2rc48" event={"ID":"182ffcc1-816e-4871-a357-6b9ac07e4481","Type":"ContainerStarted","Data":"407af4968868a1e049dc67a13340908d132b34879a8950c923b38449c409ea3d"} Sep 07 16:24:24.632178 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:24.632137 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:24:24.632500 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:24.632206 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:24:25.609331 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:25.609294 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2rc48" event={"ID":"182ffcc1-816e-4871-a357-6b9ac07e4481","Type":"ContainerStarted","Data":"12f3bab9753d39f4abd8081cbdd30573e44a0013a8bc629239494f81ecef774f"} Sep 07 16:24:25.609501 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:25.609447 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2rc48" Sep 07 16:24:25.611124 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:25.611101 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2rc48" Sep 07 16:24:25.653125 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:25.653065 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2rc48" podStartSLOduration=2.21120233 podStartE2EDuration="4.653015206s" totalImagesPullingTime="2.441812876s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:24:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:24:25.650662376 +0000 UTC m=+294.685885446" watchObservedRunningTime="2026-09-07 16:24:25.653015206 +0000 UTC m=+294.688238272" Sep 07 16:24:31.371541 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:24:31.371516 2580 kubelet.go:1648] "Image garbage collection succeeded" Sep 07 16:27:52.398451 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:27:52.398414 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-wpflr"] Sep 07 16:27:52.401620 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:27:52.401604 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wpflr" Sep 07 16:27:52.404643 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:27:52.404612 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 07 16:27:52.404643 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:27:52.404636 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-nkfbn\"" Sep 07 16:27:52.411941 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:27:52.411918 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-wpflr"] Sep 07 16:27:52.483987 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:27:52.483936 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22z9l\" (UniqueName: \"kubernetes.io/projected/174d0f3e-5f11-4428-8839-784bff454949-kube-api-access-22z9l\") pod \"payload-pre-processing-59fbc47bf8-wpflr\" (UID: \"174d0f3e-5f11-4428-8839-784bff454949\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wpflr" Sep 07 16:27:52.483987 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:27:52.484002 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/174d0f3e-5f11-4428-8839-784bff454949-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-wpflr\" (UID: \"174d0f3e-5f11-4428-8839-784bff454949\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wpflr" Sep 07 16:27:52.585119 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:27:52.585085 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22z9l\" (UniqueName: \"kubernetes.io/projected/174d0f3e-5f11-4428-8839-784bff454949-kube-api-access-22z9l\") pod \"payload-pre-processing-59fbc47bf8-wpflr\" (UID: \"174d0f3e-5f11-4428-8839-784bff454949\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wpflr" Sep 07 16:27:52.585312 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:27:52.585138 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/174d0f3e-5f11-4428-8839-784bff454949-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-wpflr\" (UID: \"174d0f3e-5f11-4428-8839-784bff454949\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wpflr" Sep 07 16:27:52.585858 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:27:52.585837 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/174d0f3e-5f11-4428-8839-784bff454949-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-wpflr\" (UID: \"174d0f3e-5f11-4428-8839-784bff454949\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wpflr" Sep 07 16:27:52.595528 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:27:52.595501 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22z9l\" (UniqueName: \"kubernetes.io/projected/174d0f3e-5f11-4428-8839-784bff454949-kube-api-access-22z9l\") pod \"payload-pre-processing-59fbc47bf8-wpflr\" (UID: \"174d0f3e-5f11-4428-8839-784bff454949\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wpflr" Sep 07 16:27:52.712727 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:27:52.712689 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wpflr" Sep 07 16:27:52.980100 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:27:52.980074 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-wpflr"] Sep 07 16:27:52.982780 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:27:52.982757 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod174d0f3e_5f11_4428_8839_784bff454949.slice/crio-187d934c5c83ce5841d4e66f39b635461542b954c17cce129900f28b7f139eb7 WatchSource:0}: Error finding container 187d934c5c83ce5841d4e66f39b635461542b954c17cce129900f28b7f139eb7: Status 404 returned error can't find the container with id 187d934c5c83ce5841d4e66f39b635461542b954c17cce129900f28b7f139eb7 Sep 07 16:27:52.985490 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:27:52.985472 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 07 16:27:53.421894 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:27:53.421818 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wpflr" event={"ID":"174d0f3e-5f11-4428-8839-784bff454949","Type":"ContainerStarted","Data":"187d934c5c83ce5841d4e66f39b635461542b954c17cce129900f28b7f139eb7"} Sep 07 16:27:55.435349 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:27:55.435318 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wpflr" event={"ID":"174d0f3e-5f11-4428-8839-784bff454949","Type":"ContainerStarted","Data":"72361b4a191f97090c5ffaf1f61949986dda498f33df82646f898e91d72b1218"} Sep 07 16:27:55.435685 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:27:55.435440 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wpflr" Sep 07 16:27:55.453331 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:27:55.453274 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wpflr" podStartSLOduration=1.08359045 podStartE2EDuration="3.453257939s" totalImagesPullingTime="2.369667489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:27:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:27:55.451967938 +0000 UTC m=+504.487191006" watchObservedRunningTime="2026-09-07 16:27:55.453257939 +0000 UTC m=+504.488481046" Sep 07 16:28:06.441737 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:28:06.441704 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wpflr" Sep 07 16:28:48.931942 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:28:48.931859 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mmfjx"] Sep 07 16:28:48.939644 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:28:48.939617 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mmfjx" Sep 07 16:28:48.943630 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:28:48.943603 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Sep 07 16:28:48.943776 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:28:48.943653 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 07 16:28:48.943776 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:28:48.943655 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 07 16:28:48.943776 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:28:48.943615 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-stl6r\"" Sep 07 16:28:48.949630 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:28:48.949608 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mmfjx"] Sep 07 16:28:49.036168 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:28:49.036136 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b11b65ed-c657-42bd-9d79-d641d429f300-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mmfjx\" (UID: \"b11b65ed-c657-42bd-9d79-d641d429f300\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mmfjx" Sep 07 16:28:49.036358 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:28:49.036180 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b11b65ed-c657-42bd-9d79-d641d429f300-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mmfjx\" (UID: \"b11b65ed-c657-42bd-9d79-d641d429f300\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mmfjx" Sep 07 16:28:49.036358 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:28:49.036201 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqnbh\" (UniqueName: \"kubernetes.io/projected/b11b65ed-c657-42bd-9d79-d641d429f300-kube-api-access-pqnbh\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mmfjx\" (UID: \"b11b65ed-c657-42bd-9d79-d641d429f300\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mmfjx" Sep 07 16:28:49.036358 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:28:49.036277 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b11b65ed-c657-42bd-9d79-d641d429f300-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mmfjx\" (UID: \"b11b65ed-c657-42bd-9d79-d641d429f300\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mmfjx" Sep 07 16:28:49.036358 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:28:49.036345 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b11b65ed-c657-42bd-9d79-d641d429f300-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mmfjx\" (UID: \"b11b65ed-c657-42bd-9d79-d641d429f300\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mmfjx" Sep 07 16:28:49.036539 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:28:49.036384 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b11b65ed-c657-42bd-9d79-d641d429f300-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mmfjx\" (UID: \"b11b65ed-c657-42bd-9d79-d641d429f300\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mmfjx" Sep 07 16:28:49.137479 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:28:49.137438 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b11b65ed-c657-42bd-9d79-d641d429f300-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mmfjx\" (UID: \"b11b65ed-c657-42bd-9d79-d641d429f300\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mmfjx" Sep 07 16:28:49.137685 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:28:49.137497 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b11b65ed-c657-42bd-9d79-d641d429f300-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mmfjx\" (UID: \"b11b65ed-c657-42bd-9d79-d641d429f300\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mmfjx" Sep 07 16:28:49.137685 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:28:49.137524 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqnbh\" (UniqueName: \"kubernetes.io/projected/b11b65ed-c657-42bd-9d79-d641d429f300-kube-api-access-pqnbh\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mmfjx\" (UID: \"b11b65ed-c657-42bd-9d79-d641d429f300\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mmfjx" Sep 07 16:28:49.137685 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:28:49.137546 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b11b65ed-c657-42bd-9d79-d641d429f300-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mmfjx\" (UID: \"b11b65ed-c657-42bd-9d79-d641d429f300\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mmfjx" Sep 07 16:28:49.137685 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:28:49.137586 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b11b65ed-c657-42bd-9d79-d641d429f300-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mmfjx\" (UID: \"b11b65ed-c657-42bd-9d79-d641d429f300\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mmfjx" Sep 07 16:28:49.137685 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:28:49.137621 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b11b65ed-c657-42bd-9d79-d641d429f300-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mmfjx\" (UID: \"b11b65ed-c657-42bd-9d79-d641d429f300\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mmfjx" Sep 07 16:28:49.137970 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:28:49.137940 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b11b65ed-c657-42bd-9d79-d641d429f300-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mmfjx\" (UID: \"b11b65ed-c657-42bd-9d79-d641d429f300\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mmfjx" Sep 07 16:28:49.138067 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:28:49.137977 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b11b65ed-c657-42bd-9d79-d641d429f300-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mmfjx\" (UID: \"b11b65ed-c657-42bd-9d79-d641d429f300\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mmfjx" Sep 07 16:28:49.138147 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:28:49.138127 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b11b65ed-c657-42bd-9d79-d641d429f300-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mmfjx\" (UID: \"b11b65ed-c657-42bd-9d79-d641d429f300\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mmfjx" Sep 07 16:28:49.140040 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:28:49.140003 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b11b65ed-c657-42bd-9d79-d641d429f300-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mmfjx\" (UID: \"b11b65ed-c657-42bd-9d79-d641d429f300\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mmfjx" Sep 07 16:28:49.140132 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:28:49.140115 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b11b65ed-c657-42bd-9d79-d641d429f300-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mmfjx\" (UID: \"b11b65ed-c657-42bd-9d79-d641d429f300\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mmfjx" Sep 07 16:28:49.146385 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:28:49.146367 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqnbh\" (UniqueName: \"kubernetes.io/projected/b11b65ed-c657-42bd-9d79-d641d429f300-kube-api-access-pqnbh\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mmfjx\" (UID: \"b11b65ed-c657-42bd-9d79-d641d429f300\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mmfjx" Sep 07 16:28:49.250380 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:28:49.250335 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mmfjx" Sep 07 16:28:49.524987 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:28:49.524913 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mmfjx"] Sep 07 16:28:49.528063 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:28:49.528014 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb11b65ed_c657_42bd_9d79_d641d429f300.slice/crio-2029e5fdb0c31d15a100fdd99e80194028f1625340c0b6f524826c564c7844d2 WatchSource:0}: Error finding container 2029e5fdb0c31d15a100fdd99e80194028f1625340c0b6f524826c564c7844d2: Status 404 returned error can't find the container with id 2029e5fdb0c31d15a100fdd99e80194028f1625340c0b6f524826c564c7844d2 Sep 07 16:28:49.645690 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:28:49.645658 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mmfjx" event={"ID":"b11b65ed-c657-42bd-9d79-d641d429f300","Type":"ContainerStarted","Data":"2029e5fdb0c31d15a100fdd99e80194028f1625340c0b6f524826c564c7844d2"} Sep 07 16:28:50.651982 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:28:50.651941 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mmfjx" event={"ID":"b11b65ed-c657-42bd-9d79-d641d429f300","Type":"ContainerStarted","Data":"daa16472d1dea993c3c23f033a96152ddef2a3ad7655d93fd6f9b8cfc7a832b8"} Sep 07 16:28:50.671696 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:28:50.671638 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mmfjx" podStartSLOduration=1.7294586509999998 podStartE2EDuration="2.671617128s" totalImagesPullingTime="942.158477ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:28:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:28:50.670523635 +0000 UTC m=+559.705746708" watchObservedRunningTime="2026-09-07 16:28:50.671617128 +0000 UTC m=+559.706840200" Sep 07 16:28:59.250724 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:28:59.250676 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mmfjx" Sep 07 16:28:59.251178 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:28:59.250859 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mmfjx" Sep 07 16:28:59.268443 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:28:59.268398 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mmfjx" Sep 07 16:28:59.699156 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:28:59.699130 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mmfjx" Sep 07 16:29:04.459894 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:04.459861 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-rlkbs"] Sep 07 16:29:04.462997 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:04.462975 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rlkbs" Sep 07 16:29:04.465861 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:04.465841 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Sep 07 16:29:04.474159 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:04.474134 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-rlkbs"] Sep 07 16:29:04.568934 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:04.568898 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a500950d-e0e6-4542-9f0a-b388e5f09b76-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rlkbs\" (UID: \"a500950d-e0e6-4542-9f0a-b388e5f09b76\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rlkbs" Sep 07 16:29:04.568934 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:04.568948 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a500950d-e0e6-4542-9f0a-b388e5f09b76-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rlkbs\" (UID: \"a500950d-e0e6-4542-9f0a-b388e5f09b76\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rlkbs" Sep 07 16:29:04.569235 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:04.569062 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44khr\" (UniqueName: \"kubernetes.io/projected/a500950d-e0e6-4542-9f0a-b388e5f09b76-kube-api-access-44khr\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rlkbs\" (UID: \"a500950d-e0e6-4542-9f0a-b388e5f09b76\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rlkbs" Sep 07 16:29:04.569235 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:04.569093 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a500950d-e0e6-4542-9f0a-b388e5f09b76-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rlkbs\" (UID: \"a500950d-e0e6-4542-9f0a-b388e5f09b76\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rlkbs" Sep 07 16:29:04.569235 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:04.569113 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a500950d-e0e6-4542-9f0a-b388e5f09b76-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rlkbs\" (UID: \"a500950d-e0e6-4542-9f0a-b388e5f09b76\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rlkbs" Sep 07 16:29:04.569235 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:04.569143 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a500950d-e0e6-4542-9f0a-b388e5f09b76-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rlkbs\" (UID: \"a500950d-e0e6-4542-9f0a-b388e5f09b76\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rlkbs" Sep 07 16:29:04.670657 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:04.670615 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44khr\" (UniqueName: \"kubernetes.io/projected/a500950d-e0e6-4542-9f0a-b388e5f09b76-kube-api-access-44khr\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rlkbs\" (UID: \"a500950d-e0e6-4542-9f0a-b388e5f09b76\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rlkbs" Sep 07 16:29:04.670858 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:04.670665 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a500950d-e0e6-4542-9f0a-b388e5f09b76-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rlkbs\" (UID: \"a500950d-e0e6-4542-9f0a-b388e5f09b76\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rlkbs" Sep 07 16:29:04.670858 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:04.670695 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a500950d-e0e6-4542-9f0a-b388e5f09b76-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rlkbs\" (UID: \"a500950d-e0e6-4542-9f0a-b388e5f09b76\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rlkbs" Sep 07 16:29:04.670858 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:04.670722 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a500950d-e0e6-4542-9f0a-b388e5f09b76-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rlkbs\" (UID: \"a500950d-e0e6-4542-9f0a-b388e5f09b76\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rlkbs" Sep 07 16:29:04.670858 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:04.670777 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a500950d-e0e6-4542-9f0a-b388e5f09b76-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rlkbs\" (UID: \"a500950d-e0e6-4542-9f0a-b388e5f09b76\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rlkbs" Sep 07 16:29:04.670858 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:04.670816 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a500950d-e0e6-4542-9f0a-b388e5f09b76-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rlkbs\" (UID: \"a500950d-e0e6-4542-9f0a-b388e5f09b76\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rlkbs" Sep 07 16:29:04.671202 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:04.671177 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a500950d-e0e6-4542-9f0a-b388e5f09b76-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rlkbs\" (UID: \"a500950d-e0e6-4542-9f0a-b388e5f09b76\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rlkbs" Sep 07 16:29:04.671275 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:04.671188 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a500950d-e0e6-4542-9f0a-b388e5f09b76-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rlkbs\" (UID: \"a500950d-e0e6-4542-9f0a-b388e5f09b76\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rlkbs" Sep 07 16:29:04.671275 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:04.671256 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a500950d-e0e6-4542-9f0a-b388e5f09b76-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rlkbs\" (UID: \"a500950d-e0e6-4542-9f0a-b388e5f09b76\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rlkbs" Sep 07 16:29:04.673319 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:04.673300 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a500950d-e0e6-4542-9f0a-b388e5f09b76-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rlkbs\" (UID: \"a500950d-e0e6-4542-9f0a-b388e5f09b76\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rlkbs" Sep 07 16:29:04.673439 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:04.673424 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a500950d-e0e6-4542-9f0a-b388e5f09b76-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rlkbs\" (UID: \"a500950d-e0e6-4542-9f0a-b388e5f09b76\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rlkbs" Sep 07 16:29:04.679036 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:04.678998 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44khr\" (UniqueName: \"kubernetes.io/projected/a500950d-e0e6-4542-9f0a-b388e5f09b76-kube-api-access-44khr\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rlkbs\" (UID: \"a500950d-e0e6-4542-9f0a-b388e5f09b76\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rlkbs" Sep 07 16:29:04.773819 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:04.773736 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rlkbs" Sep 07 16:29:05.065822 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:05.065793 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-rlkbs"] Sep 07 16:29:05.067131 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:29:05.067103 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda500950d_e0e6_4542_9f0a_b388e5f09b76.slice/crio-2e169dbe81262e4c98966c41e566ab88ae2eb693d2d7fe06dca68ae9d0310658 WatchSource:0}: Error finding container 2e169dbe81262e4c98966c41e566ab88ae2eb693d2d7fe06dca68ae9d0310658: Status 404 returned error can't find the container with id 2e169dbe81262e4c98966c41e566ab88ae2eb693d2d7fe06dca68ae9d0310658 Sep 07 16:29:05.713446 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:05.713407 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rlkbs" event={"ID":"a500950d-e0e6-4542-9f0a-b388e5f09b76","Type":"ContainerStarted","Data":"cb815d54e53124f37a0149353b59e0122880a2c381d253c68281a1ef844612b5"} Sep 07 16:29:05.713832 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:05.713469 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rlkbs" event={"ID":"a500950d-e0e6-4542-9f0a-b388e5f09b76","Type":"ContainerStarted","Data":"2e169dbe81262e4c98966c41e566ab88ae2eb693d2d7fe06dca68ae9d0310658"} Sep 07 16:29:05.742075 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:05.742005 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rlkbs" podStartSLOduration=1.57305224 podStartE2EDuration="1.74198908s" totalImagesPullingTime="168.93684ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:29:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:29:05.741684987 +0000 UTC m=+574.776908056" watchObservedRunningTime="2026-09-07 16:29:05.74198908 +0000 UTC m=+574.777212159" Sep 07 16:29:14.569618 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:14.569587 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjgstf"] Sep 07 16:29:14.572823 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:14.572805 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjgstf" Sep 07 16:29:14.575329 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:14.575308 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Sep 07 16:29:14.597403 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:14.597372 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjgstf"] Sep 07 16:29:14.655977 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:14.655935 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4897fdef-8bf5-485d-8e7b-51e239ad09f4-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjgstf\" (UID: \"4897fdef-8bf5-485d-8e7b-51e239ad09f4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjgstf" Sep 07 16:29:14.655977 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:14.655982 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4897fdef-8bf5-485d-8e7b-51e239ad09f4-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjgstf\" (UID: \"4897fdef-8bf5-485d-8e7b-51e239ad09f4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjgstf" Sep 07 16:29:14.656210 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:14.656074 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4897fdef-8bf5-485d-8e7b-51e239ad09f4-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjgstf\" (UID: \"4897fdef-8bf5-485d-8e7b-51e239ad09f4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjgstf" Sep 07 16:29:14.656210 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:14.656095 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4897fdef-8bf5-485d-8e7b-51e239ad09f4-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjgstf\" (UID: \"4897fdef-8bf5-485d-8e7b-51e239ad09f4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjgstf" Sep 07 16:29:14.656210 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:14.656123 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xdg7\" (UniqueName: \"kubernetes.io/projected/4897fdef-8bf5-485d-8e7b-51e239ad09f4-kube-api-access-7xdg7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjgstf\" (UID: \"4897fdef-8bf5-485d-8e7b-51e239ad09f4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjgstf" Sep 07 16:29:14.656210 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:14.656177 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4897fdef-8bf5-485d-8e7b-51e239ad09f4-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjgstf\" (UID: \"4897fdef-8bf5-485d-8e7b-51e239ad09f4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjgstf" Sep 07 16:29:14.756993 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:14.756965 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4897fdef-8bf5-485d-8e7b-51e239ad09f4-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjgstf\" (UID: \"4897fdef-8bf5-485d-8e7b-51e239ad09f4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjgstf" Sep 07 16:29:14.757201 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:14.757003 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xdg7\" (UniqueName: \"kubernetes.io/projected/4897fdef-8bf5-485d-8e7b-51e239ad09f4-kube-api-access-7xdg7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjgstf\" (UID: \"4897fdef-8bf5-485d-8e7b-51e239ad09f4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjgstf" Sep 07 16:29:14.757201 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:14.757059 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4897fdef-8bf5-485d-8e7b-51e239ad09f4-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjgstf\" (UID: \"4897fdef-8bf5-485d-8e7b-51e239ad09f4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjgstf" Sep 07 16:29:14.757201 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:14.757118 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4897fdef-8bf5-485d-8e7b-51e239ad09f4-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjgstf\" (UID: \"4897fdef-8bf5-485d-8e7b-51e239ad09f4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjgstf" Sep 07 16:29:14.757321 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:14.757244 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4897fdef-8bf5-485d-8e7b-51e239ad09f4-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjgstf\" (UID: \"4897fdef-8bf5-485d-8e7b-51e239ad09f4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjgstf" Sep 07 16:29:14.757363 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:14.757328 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4897fdef-8bf5-485d-8e7b-51e239ad09f4-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjgstf\" (UID: \"4897fdef-8bf5-485d-8e7b-51e239ad09f4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjgstf" Sep 07 16:29:14.757460 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:14.757436 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4897fdef-8bf5-485d-8e7b-51e239ad09f4-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjgstf\" (UID: \"4897fdef-8bf5-485d-8e7b-51e239ad09f4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjgstf" Sep 07 16:29:14.757517 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:14.757492 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4897fdef-8bf5-485d-8e7b-51e239ad09f4-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjgstf\" (UID: \"4897fdef-8bf5-485d-8e7b-51e239ad09f4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjgstf" Sep 07 16:29:14.757615 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:14.757600 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4897fdef-8bf5-485d-8e7b-51e239ad09f4-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjgstf\" (UID: \"4897fdef-8bf5-485d-8e7b-51e239ad09f4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjgstf" Sep 07 16:29:14.759539 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:14.759520 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4897fdef-8bf5-485d-8e7b-51e239ad09f4-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjgstf\" (UID: \"4897fdef-8bf5-485d-8e7b-51e239ad09f4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjgstf" Sep 07 16:29:14.759725 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:14.759664 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4897fdef-8bf5-485d-8e7b-51e239ad09f4-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjgstf\" (UID: \"4897fdef-8bf5-485d-8e7b-51e239ad09f4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjgstf" Sep 07 16:29:14.768109 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:14.768084 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xdg7\" (UniqueName: \"kubernetes.io/projected/4897fdef-8bf5-485d-8e7b-51e239ad09f4-kube-api-access-7xdg7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjgstf\" (UID: \"4897fdef-8bf5-485d-8e7b-51e239ad09f4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjgstf" Sep 07 16:29:14.773882 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:14.773859 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rlkbs" Sep 07 16:29:14.773987 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:14.773921 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rlkbs" Sep 07 16:29:14.786251 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:14.786221 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rlkbs" Sep 07 16:29:14.882580 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:14.882492 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjgstf" Sep 07 16:29:15.165637 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:15.165545 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjgstf"] Sep 07 16:29:15.169462 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:29:15.169437 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4897fdef_8bf5_485d_8e7b_51e239ad09f4.slice/crio-f10667487386adbbf384522a428496690ee0b89ea29cd3b6b86bd74ee8ae3347 WatchSource:0}: Error finding container f10667487386adbbf384522a428496690ee0b89ea29cd3b6b86bd74ee8ae3347: Status 404 returned error can't find the container with id f10667487386adbbf384522a428496690ee0b89ea29cd3b6b86bd74ee8ae3347 Sep 07 16:29:15.755448 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:15.755409 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjgstf" event={"ID":"4897fdef-8bf5-485d-8e7b-51e239ad09f4","Type":"ContainerStarted","Data":"2b1d3878f93829ddd16639a6df5be06c9f73e4bc94d556ab502104f0490914c4"} Sep 07 16:29:15.755933 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:15.755504 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjgstf" event={"ID":"4897fdef-8bf5-485d-8e7b-51e239ad09f4","Type":"ContainerStarted","Data":"f10667487386adbbf384522a428496690ee0b89ea29cd3b6b86bd74ee8ae3347"} Sep 07 16:29:15.767136 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:15.767110 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rlkbs" Sep 07 16:29:15.788875 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:15.788822 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjgstf" podStartSLOduration=1.612770357 podStartE2EDuration="1.788807428s" totalImagesPullingTime="176.037071ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:29:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:29:15.7861202 +0000 UTC m=+584.821343271" watchObservedRunningTime="2026-09-07 16:29:15.788807428 +0000 UTC m=+584.824030498" Sep 07 16:29:23.830301 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:23.830252 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-lxzg7"] Sep 07 16:29:23.834498 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:23.834477 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lxzg7" Sep 07 16:29:23.836941 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:23.836918 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Sep 07 16:29:23.844176 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:23.844151 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-lxzg7"] Sep 07 16:29:23.934491 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:23.934455 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c26734a-083f-468f-8f8d-a3ced9e3c898-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lxzg7\" (UID: \"1c26734a-083f-468f-8f8d-a3ced9e3c898\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lxzg7" Sep 07 16:29:23.934651 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:23.934511 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c26734a-083f-468f-8f8d-a3ced9e3c898-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lxzg7\" (UID: \"1c26734a-083f-468f-8f8d-a3ced9e3c898\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lxzg7" Sep 07 16:29:23.934651 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:23.934571 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c26734a-083f-468f-8f8d-a3ced9e3c898-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lxzg7\" (UID: \"1c26734a-083f-468f-8f8d-a3ced9e3c898\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lxzg7" Sep 07 16:29:23.934651 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:23.934620 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c26734a-083f-468f-8f8d-a3ced9e3c898-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lxzg7\" (UID: \"1c26734a-083f-468f-8f8d-a3ced9e3c898\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lxzg7" Sep 07 16:29:23.934749 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:23.934693 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c26734a-083f-468f-8f8d-a3ced9e3c898-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lxzg7\" (UID: \"1c26734a-083f-468f-8f8d-a3ced9e3c898\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lxzg7" Sep 07 16:29:23.934749 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:23.934714 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dr66\" (UniqueName: \"kubernetes.io/projected/1c26734a-083f-468f-8f8d-a3ced9e3c898-kube-api-access-5dr66\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lxzg7\" (UID: \"1c26734a-083f-468f-8f8d-a3ced9e3c898\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lxzg7" Sep 07 16:29:24.035868 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:24.035828 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c26734a-083f-468f-8f8d-a3ced9e3c898-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lxzg7\" (UID: \"1c26734a-083f-468f-8f8d-a3ced9e3c898\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lxzg7" Sep 07 16:29:24.035868 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:24.035869 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dr66\" (UniqueName: \"kubernetes.io/projected/1c26734a-083f-468f-8f8d-a3ced9e3c898-kube-api-access-5dr66\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lxzg7\" (UID: \"1c26734a-083f-468f-8f8d-a3ced9e3c898\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lxzg7" Sep 07 16:29:24.036147 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:24.035888 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c26734a-083f-468f-8f8d-a3ced9e3c898-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lxzg7\" (UID: \"1c26734a-083f-468f-8f8d-a3ced9e3c898\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lxzg7" Sep 07 16:29:24.036147 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:24.035939 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c26734a-083f-468f-8f8d-a3ced9e3c898-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lxzg7\" (UID: \"1c26734a-083f-468f-8f8d-a3ced9e3c898\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lxzg7" Sep 07 16:29:24.036147 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:24.035979 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c26734a-083f-468f-8f8d-a3ced9e3c898-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lxzg7\" (UID: \"1c26734a-083f-468f-8f8d-a3ced9e3c898\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lxzg7" Sep 07 16:29:24.036147 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:24.036016 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c26734a-083f-468f-8f8d-a3ced9e3c898-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lxzg7\" (UID: \"1c26734a-083f-468f-8f8d-a3ced9e3c898\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lxzg7" Sep 07 16:29:24.036436 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:24.036411 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c26734a-083f-468f-8f8d-a3ced9e3c898-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lxzg7\" (UID: \"1c26734a-083f-468f-8f8d-a3ced9e3c898\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lxzg7" Sep 07 16:29:24.036516 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:24.036462 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c26734a-083f-468f-8f8d-a3ced9e3c898-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lxzg7\" (UID: \"1c26734a-083f-468f-8f8d-a3ced9e3c898\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lxzg7" Sep 07 16:29:24.036516 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:24.036482 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c26734a-083f-468f-8f8d-a3ced9e3c898-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lxzg7\" (UID: \"1c26734a-083f-468f-8f8d-a3ced9e3c898\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lxzg7" Sep 07 16:29:24.038357 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:24.038334 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c26734a-083f-468f-8f8d-a3ced9e3c898-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lxzg7\" (UID: \"1c26734a-083f-468f-8f8d-a3ced9e3c898\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lxzg7" Sep 07 16:29:24.038524 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:24.038507 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c26734a-083f-468f-8f8d-a3ced9e3c898-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lxzg7\" (UID: \"1c26734a-083f-468f-8f8d-a3ced9e3c898\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lxzg7" Sep 07 16:29:24.043204 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:24.043184 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dr66\" (UniqueName: \"kubernetes.io/projected/1c26734a-083f-468f-8f8d-a3ced9e3c898-kube-api-access-5dr66\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lxzg7\" (UID: \"1c26734a-083f-468f-8f8d-a3ced9e3c898\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lxzg7" Sep 07 16:29:24.145729 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:24.145648 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lxzg7" Sep 07 16:29:24.418917 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:24.418826 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-lxzg7"] Sep 07 16:29:24.421831 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:29:24.421791 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c26734a_083f_468f_8f8d_a3ced9e3c898.slice/crio-f3fd60e20dd47b3de458ddc0f67ac5c771f45ced98053ef505c021b5fb88196c WatchSource:0}: Error finding container f3fd60e20dd47b3de458ddc0f67ac5c771f45ced98053ef505c021b5fb88196c: Status 404 returned error can't find the container with id f3fd60e20dd47b3de458ddc0f67ac5c771f45ced98053ef505c021b5fb88196c Sep 07 16:29:24.792533 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:24.792487 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lxzg7" event={"ID":"1c26734a-083f-468f-8f8d-a3ced9e3c898","Type":"ContainerStarted","Data":"b2a9b86d925004d5cbe38e3f785f254be3769ecc30860036845ed83b6137c6da"} Sep 07 16:29:24.792712 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:24.792588 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lxzg7" event={"ID":"1c26734a-083f-468f-8f8d-a3ced9e3c898","Type":"ContainerStarted","Data":"f3fd60e20dd47b3de458ddc0f67ac5c771f45ced98053ef505c021b5fb88196c"} Sep 07 16:29:24.809750 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:24.809696 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lxzg7" podStartSLOduration=1.637196356 podStartE2EDuration="1.809680124s" totalImagesPullingTime="172.483768ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:29:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:29:24.808510355 +0000 UTC m=+593.843733426" watchObservedRunningTime="2026-09-07 16:29:24.809680124 +0000 UTC m=+593.844903197" Sep 07 16:29:24.882865 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:24.882825 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjgstf" Sep 07 16:29:24.883402 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:24.882897 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjgstf" Sep 07 16:29:24.896347 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:24.896323 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjgstf" Sep 07 16:29:25.808707 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:25.808671 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjgstf" Sep 07 16:29:34.146566 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:34.146523 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lxzg7" Sep 07 16:29:34.146944 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:34.146590 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lxzg7" Sep 07 16:29:34.159185 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:34.159151 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lxzg7" Sep 07 16:29:34.845077 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:29:34.845045 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lxzg7" Sep 07 16:30:50.561118 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:50.561082 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-747d6d-8d79cd766-hnpxc"] Sep 07 16:30:50.564221 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:50.564204 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-747d6d-8d79cd766-hnpxc" Sep 07 16:30:50.566485 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:50.566462 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-747d6d\"" Sep 07 16:30:50.566620 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:50.566508 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-747d6d-dockercfg-28n7g\"" Sep 07 16:30:50.574338 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:50.574312 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-747d6d-8d79cd766-hnpxc"] Sep 07 16:30:50.595532 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:50.595501 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/491162f6-cac0-42a6-b71e-1c9d976f95f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-747d6d-8d79cd766-hnpxc\" (UID: \"491162f6-cac0-42a6-b71e-1c9d976f95f9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-747d6d-8d79cd766-hnpxc" Sep 07 16:30:50.595692 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:50.595562 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4g4v\" (UniqueName: \"kubernetes.io/projected/491162f6-cac0-42a6-b71e-1c9d976f95f9-kube-api-access-q4g4v\") pod \"payload-pre-processing-e2e-inf-a-747d6d-8d79cd766-hnpxc\" (UID: \"491162f6-cac0-42a6-b71e-1c9d976f95f9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-747d6d-8d79cd766-hnpxc" Sep 07 16:30:50.696884 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:50.696842 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/491162f6-cac0-42a6-b71e-1c9d976f95f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-747d6d-8d79cd766-hnpxc\" (UID: \"491162f6-cac0-42a6-b71e-1c9d976f95f9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-747d6d-8d79cd766-hnpxc" Sep 07 16:30:50.697092 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:50.696909 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4g4v\" (UniqueName: \"kubernetes.io/projected/491162f6-cac0-42a6-b71e-1c9d976f95f9-kube-api-access-q4g4v\") pod \"payload-pre-processing-e2e-inf-a-747d6d-8d79cd766-hnpxc\" (UID: \"491162f6-cac0-42a6-b71e-1c9d976f95f9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-747d6d-8d79cd766-hnpxc" Sep 07 16:30:50.697542 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:50.697524 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/491162f6-cac0-42a6-b71e-1c9d976f95f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-747d6d-8d79cd766-hnpxc\" (UID: \"491162f6-cac0-42a6-b71e-1c9d976f95f9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-747d6d-8d79cd766-hnpxc" Sep 07 16:30:50.704390 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:50.704373 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4g4v\" (UniqueName: \"kubernetes.io/projected/491162f6-cac0-42a6-b71e-1c9d976f95f9-kube-api-access-q4g4v\") pod \"payload-pre-processing-e2e-inf-a-747d6d-8d79cd766-hnpxc\" (UID: \"491162f6-cac0-42a6-b71e-1c9d976f95f9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-747d6d-8d79cd766-hnpxc" Sep 07 16:30:50.848611 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:50.848538 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv"] Sep 07 16:30:50.852100 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:50.852072 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv" Sep 07 16:30:50.854205 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:50.854187 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-8a3403-openshift-default-dockercfg-5qcpd\"" Sep 07 16:30:50.866301 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:50.866259 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv"] Sep 07 16:30:50.876127 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:50.876105 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-747d6d-8d79cd766-hnpxc" Sep 07 16:30:50.899753 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:50.899729 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69c3fe33-98c9-404c-bccf-548c1f77e9ec-workload-certs\") pod \"e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv\" (UID: \"69c3fe33-98c9-404c-bccf-548c1f77e9ec\") " pod="openshift-ingress/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv" Sep 07 16:30:50.899856 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:50.899766 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwsrj\" (UniqueName: \"kubernetes.io/projected/69c3fe33-98c9-404c-bccf-548c1f77e9ec-kube-api-access-vwsrj\") pod \"e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv\" (UID: \"69c3fe33-98c9-404c-bccf-548c1f77e9ec\") " pod="openshift-ingress/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv" Sep 07 16:30:50.899856 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:50.899798 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69c3fe33-98c9-404c-bccf-548c1f77e9ec-credential-socket\") pod \"e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv\" (UID: \"69c3fe33-98c9-404c-bccf-548c1f77e9ec\") " pod="openshift-ingress/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv" Sep 07 16:30:50.899856 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:50.899813 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69c3fe33-98c9-404c-bccf-548c1f77e9ec-istio-envoy\") pod \"e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv\" (UID: \"69c3fe33-98c9-404c-bccf-548c1f77e9ec\") " pod="openshift-ingress/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv" Sep 07 16:30:50.899985 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:50.899906 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69c3fe33-98c9-404c-bccf-548c1f77e9ec-istio-token\") pod \"e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv\" (UID: \"69c3fe33-98c9-404c-bccf-548c1f77e9ec\") " pod="openshift-ingress/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv" Sep 07 16:30:50.899985 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:50.899930 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69c3fe33-98c9-404c-bccf-548c1f77e9ec-istio-data\") pod \"e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv\" (UID: \"69c3fe33-98c9-404c-bccf-548c1f77e9ec\") " pod="openshift-ingress/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv" Sep 07 16:30:50.899985 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:50.899946 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69c3fe33-98c9-404c-bccf-548c1f77e9ec-istio-podinfo\") pod \"e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv\" (UID: \"69c3fe33-98c9-404c-bccf-548c1f77e9ec\") " pod="openshift-ingress/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv" Sep 07 16:30:50.899985 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:50.899971 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69c3fe33-98c9-404c-bccf-548c1f77e9ec-workload-socket\") pod \"e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv\" (UID: \"69c3fe33-98c9-404c-bccf-548c1f77e9ec\") " pod="openshift-ingress/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv" Sep 07 16:30:50.900142 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:50.899993 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69c3fe33-98c9-404c-bccf-548c1f77e9ec-istiod-ca-cert\") pod \"e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv\" (UID: \"69c3fe33-98c9-404c-bccf-548c1f77e9ec\") " pod="openshift-ingress/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv" Sep 07 16:30:51.000750 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:51.000714 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69c3fe33-98c9-404c-bccf-548c1f77e9ec-credential-socket\") pod \"e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv\" (UID: \"69c3fe33-98c9-404c-bccf-548c1f77e9ec\") " pod="openshift-ingress/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv" Sep 07 16:30:51.000750 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:51.000752 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69c3fe33-98c9-404c-bccf-548c1f77e9ec-istio-envoy\") pod \"e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv\" (UID: \"69c3fe33-98c9-404c-bccf-548c1f77e9ec\") " pod="openshift-ingress/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv" Sep 07 16:30:51.000973 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:51.000772 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69c3fe33-98c9-404c-bccf-548c1f77e9ec-istio-token\") pod \"e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv\" (UID: \"69c3fe33-98c9-404c-bccf-548c1f77e9ec\") " pod="openshift-ingress/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv" Sep 07 16:30:51.000973 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:51.000800 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69c3fe33-98c9-404c-bccf-548c1f77e9ec-istio-data\") pod \"e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv\" (UID: \"69c3fe33-98c9-404c-bccf-548c1f77e9ec\") " pod="openshift-ingress/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv" Sep 07 16:30:51.000973 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:51.000828 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69c3fe33-98c9-404c-bccf-548c1f77e9ec-istio-podinfo\") pod \"e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv\" (UID: \"69c3fe33-98c9-404c-bccf-548c1f77e9ec\") " pod="openshift-ingress/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv" Sep 07 16:30:51.000973 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:51.000871 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69c3fe33-98c9-404c-bccf-548c1f77e9ec-workload-socket\") pod \"e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv\" (UID: \"69c3fe33-98c9-404c-bccf-548c1f77e9ec\") " pod="openshift-ingress/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv" Sep 07 16:30:51.000973 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:51.000899 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69c3fe33-98c9-404c-bccf-548c1f77e9ec-istiod-ca-cert\") pod \"e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv\" (UID: \"69c3fe33-98c9-404c-bccf-548c1f77e9ec\") " pod="openshift-ingress/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv" Sep 07 16:30:51.001247 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:51.001190 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69c3fe33-98c9-404c-bccf-548c1f77e9ec-workload-certs\") pod \"e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv\" (UID: \"69c3fe33-98c9-404c-bccf-548c1f77e9ec\") " pod="openshift-ingress/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv" Sep 07 16:30:51.001247 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:51.001216 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69c3fe33-98c9-404c-bccf-548c1f77e9ec-credential-socket\") pod \"e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv\" (UID: \"69c3fe33-98c9-404c-bccf-548c1f77e9ec\") " pod="openshift-ingress/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv" Sep 07 16:30:51.001247 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:51.001235 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwsrj\" (UniqueName: \"kubernetes.io/projected/69c3fe33-98c9-404c-bccf-548c1f77e9ec-kube-api-access-vwsrj\") pod \"e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv\" (UID: \"69c3fe33-98c9-404c-bccf-548c1f77e9ec\") " pod="openshift-ingress/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv" Sep 07 16:30:51.001719 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:51.001673 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69c3fe33-98c9-404c-bccf-548c1f77e9ec-workload-socket\") pod \"e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv\" (UID: \"69c3fe33-98c9-404c-bccf-548c1f77e9ec\") " pod="openshift-ingress/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv" Sep 07 16:30:51.001719 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:51.001692 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69c3fe33-98c9-404c-bccf-548c1f77e9ec-istio-data\") pod \"e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv\" (UID: \"69c3fe33-98c9-404c-bccf-548c1f77e9ec\") " pod="openshift-ingress/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv" Sep 07 16:30:51.001872 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:51.001848 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69c3fe33-98c9-404c-bccf-548c1f77e9ec-workload-certs\") pod \"e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv\" (UID: \"69c3fe33-98c9-404c-bccf-548c1f77e9ec\") " pod="openshift-ingress/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv" Sep 07 16:30:51.002104 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:51.002085 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69c3fe33-98c9-404c-bccf-548c1f77e9ec-istiod-ca-cert\") pod \"e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv\" (UID: \"69c3fe33-98c9-404c-bccf-548c1f77e9ec\") " pod="openshift-ingress/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv" Sep 07 16:30:51.003382 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:51.003358 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69c3fe33-98c9-404c-bccf-548c1f77e9ec-istio-envoy\") pod \"e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv\" (UID: \"69c3fe33-98c9-404c-bccf-548c1f77e9ec\") " pod="openshift-ingress/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv" Sep 07 16:30:51.004039 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:51.004001 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69c3fe33-98c9-404c-bccf-548c1f77e9ec-istio-podinfo\") pod \"e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv\" (UID: \"69c3fe33-98c9-404c-bccf-548c1f77e9ec\") " pod="openshift-ingress/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv" Sep 07 16:30:51.008399 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:51.008380 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69c3fe33-98c9-404c-bccf-548c1f77e9ec-istio-token\") pod \"e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv\" (UID: \"69c3fe33-98c9-404c-bccf-548c1f77e9ec\") " pod="openshift-ingress/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv" Sep 07 16:30:51.008503 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:51.008409 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwsrj\" (UniqueName: \"kubernetes.io/projected/69c3fe33-98c9-404c-bccf-548c1f77e9ec-kube-api-access-vwsrj\") pod \"e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv\" (UID: \"69c3fe33-98c9-404c-bccf-548c1f77e9ec\") " pod="openshift-ingress/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv" Sep 07 16:30:51.156124 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:51.155979 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-747d6d-8d79cd766-hnpxc"] Sep 07 16:30:51.158709 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:30:51.158682 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod491162f6_cac0_42a6_b71e_1c9d976f95f9.slice/crio-d6f13e202dae9ec38fd244d5979a0a497e13bf1aefeed4219659135f6ce76d07 WatchSource:0}: Error finding container d6f13e202dae9ec38fd244d5979a0a497e13bf1aefeed4219659135f6ce76d07: Status 404 returned error can't find the container with id d6f13e202dae9ec38fd244d5979a0a497e13bf1aefeed4219659135f6ce76d07 Sep 07 16:30:51.164584 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:51.164563 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv" Sep 07 16:30:51.449841 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:51.449814 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv"] Sep 07 16:30:51.451313 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:30:51.451289 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69c3fe33_98c9_404c_bccf_548c1f77e9ec.slice/crio-21a2a803c9f4292cce8b9ab61620f7272cd49c85661f006594d380cab52575fe WatchSource:0}: Error finding container 21a2a803c9f4292cce8b9ab61620f7272cd49c85661f006594d380cab52575fe: Status 404 returned error can't find the container with id 21a2a803c9f4292cce8b9ab61620f7272cd49c85661f006594d380cab52575fe Sep 07 16:30:51.453635 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:51.453590 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:30:51.453732 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:51.453685 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:30:51.453789 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:51.453731 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:30:52.062458 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:52.062424 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8"] Sep 07 16:30:52.066204 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:52.066180 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8" Sep 07 16:30:52.068415 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:52.068394 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-983d9f00-openshift-default-dockercfg-lpbjh\"" Sep 07 16:30:52.078180 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:52.078150 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8"] Sep 07 16:30:52.111438 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:52.111387 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6235888e-5e89-4892-b007-701682139ebd-istio-podinfo\") pod \"e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8\" (UID: \"6235888e-5e89-4892-b007-701682139ebd\") " pod="openshift-ingress/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8" Sep 07 16:30:52.111626 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:52.111443 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6235888e-5e89-4892-b007-701682139ebd-istio-data\") pod \"e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8\" (UID: \"6235888e-5e89-4892-b007-701682139ebd\") " pod="openshift-ingress/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8" Sep 07 16:30:52.111626 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:52.111470 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6235888e-5e89-4892-b007-701682139ebd-istiod-ca-cert\") pod \"e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8\" (UID: \"6235888e-5e89-4892-b007-701682139ebd\") " pod="openshift-ingress/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8" Sep 07 16:30:52.111626 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:52.111527 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdztr\" (UniqueName: \"kubernetes.io/projected/6235888e-5e89-4892-b007-701682139ebd-kube-api-access-jdztr\") pod \"e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8\" (UID: \"6235888e-5e89-4892-b007-701682139ebd\") " pod="openshift-ingress/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8" Sep 07 16:30:52.111626 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:52.111563 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6235888e-5e89-4892-b007-701682139ebd-workload-socket\") pod \"e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8\" (UID: \"6235888e-5e89-4892-b007-701682139ebd\") " pod="openshift-ingress/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8" Sep 07 16:30:52.111829 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:52.111651 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6235888e-5e89-4892-b007-701682139ebd-credential-socket\") pod \"e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8\" (UID: \"6235888e-5e89-4892-b007-701682139ebd\") " pod="openshift-ingress/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8" Sep 07 16:30:52.111829 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:52.111682 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6235888e-5e89-4892-b007-701682139ebd-istio-envoy\") pod \"e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8\" (UID: \"6235888e-5e89-4892-b007-701682139ebd\") " pod="openshift-ingress/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8" Sep 07 16:30:52.111829 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:52.111706 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6235888e-5e89-4892-b007-701682139ebd-istio-token\") pod \"e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8\" (UID: \"6235888e-5e89-4892-b007-701682139ebd\") " pod="openshift-ingress/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8" Sep 07 16:30:52.111829 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:52.111805 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6235888e-5e89-4892-b007-701682139ebd-workload-certs\") pod \"e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8\" (UID: \"6235888e-5e89-4892-b007-701682139ebd\") " pod="openshift-ingress/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8" Sep 07 16:30:52.171667 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:52.170952 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv" event={"ID":"69c3fe33-98c9-404c-bccf-548c1f77e9ec","Type":"ContainerStarted","Data":"72fe1639848a9b30d82f8423089b7e990726c8920cef18a6690012cd9de3a842"} Sep 07 16:30:52.171667 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:52.171070 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv" event={"ID":"69c3fe33-98c9-404c-bccf-548c1f77e9ec","Type":"ContainerStarted","Data":"21a2a803c9f4292cce8b9ab61620f7272cd49c85661f006594d380cab52575fe"} Sep 07 16:30:52.173453 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:52.173418 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-747d6d-8d79cd766-hnpxc" event={"ID":"491162f6-cac0-42a6-b71e-1c9d976f95f9","Type":"ContainerStarted","Data":"686fdd8cc88cf1e12e2201310cf2cfc548655f4a30aa2f8b5f5715555153d006"} Sep 07 16:30:52.173585 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:52.173472 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-747d6d-8d79cd766-hnpxc" event={"ID":"491162f6-cac0-42a6-b71e-1c9d976f95f9","Type":"ContainerStarted","Data":"d6f13e202dae9ec38fd244d5979a0a497e13bf1aefeed4219659135f6ce76d07"} Sep 07 16:30:52.173585 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:52.173518 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-747d6d-8d79cd766-hnpxc" Sep 07 16:30:52.188018 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:52.187955 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv" podStartSLOduration=2.187894714 podStartE2EDuration="2.187894714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:30:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:30:52.187811475 +0000 UTC m=+681.223034546" watchObservedRunningTime="2026-09-07 16:30:52.187894714 +0000 UTC m=+681.223117785" Sep 07 16:30:52.202410 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:52.202349 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-747d6d-8d79cd766-hnpxc" podStartSLOduration=2.202328443 podStartE2EDuration="2.202328443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:30:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:30:52.201384009 +0000 UTC m=+681.236607078" watchObservedRunningTime="2026-09-07 16:30:52.202328443 +0000 UTC m=+681.237551515" Sep 07 16:30:52.213052 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:52.213002 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6235888e-5e89-4892-b007-701682139ebd-credential-socket\") pod \"e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8\" (UID: \"6235888e-5e89-4892-b007-701682139ebd\") " pod="openshift-ingress/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8" Sep 07 16:30:52.213052 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:52.213049 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6235888e-5e89-4892-b007-701682139ebd-istio-envoy\") pod \"e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8\" (UID: \"6235888e-5e89-4892-b007-701682139ebd\") " pod="openshift-ingress/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8" Sep 07 16:30:52.213261 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:52.213067 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6235888e-5e89-4892-b007-701682139ebd-istio-token\") pod \"e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8\" (UID: \"6235888e-5e89-4892-b007-701682139ebd\") " pod="openshift-ingress/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8" Sep 07 16:30:52.213261 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:52.213166 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6235888e-5e89-4892-b007-701682139ebd-workload-certs\") pod \"e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8\" (UID: \"6235888e-5e89-4892-b007-701682139ebd\") " pod="openshift-ingress/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8" Sep 07 16:30:52.213261 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:52.213240 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6235888e-5e89-4892-b007-701682139ebd-istio-podinfo\") pod \"e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8\" (UID: \"6235888e-5e89-4892-b007-701682139ebd\") " pod="openshift-ingress/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8" Sep 07 16:30:52.213411 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:52.213270 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6235888e-5e89-4892-b007-701682139ebd-istio-data\") pod \"e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8\" (UID: \"6235888e-5e89-4892-b007-701682139ebd\") " pod="openshift-ingress/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8" Sep 07 16:30:52.213411 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:52.213295 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6235888e-5e89-4892-b007-701682139ebd-istiod-ca-cert\") pod \"e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8\" (UID: \"6235888e-5e89-4892-b007-701682139ebd\") " pod="openshift-ingress/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8" Sep 07 16:30:52.213411 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:52.213328 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdztr\" (UniqueName: \"kubernetes.io/projected/6235888e-5e89-4892-b007-701682139ebd-kube-api-access-jdztr\") pod \"e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8\" (UID: \"6235888e-5e89-4892-b007-701682139ebd\") " pod="openshift-ingress/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8" Sep 07 16:30:52.213411 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:52.213368 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6235888e-5e89-4892-b007-701682139ebd-workload-socket\") pod \"e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8\" (UID: \"6235888e-5e89-4892-b007-701682139ebd\") " pod="openshift-ingress/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8" Sep 07 16:30:52.213606 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:52.213422 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6235888e-5e89-4892-b007-701682139ebd-credential-socket\") pod \"e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8\" (UID: \"6235888e-5e89-4892-b007-701682139ebd\") " pod="openshift-ingress/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8" Sep 07 16:30:52.213671 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:52.213641 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6235888e-5e89-4892-b007-701682139ebd-workload-socket\") pod \"e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8\" (UID: \"6235888e-5e89-4892-b007-701682139ebd\") " pod="openshift-ingress/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8" Sep 07 16:30:52.213733 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:52.213694 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6235888e-5e89-4892-b007-701682139ebd-workload-certs\") pod \"e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8\" (UID: \"6235888e-5e89-4892-b007-701682139ebd\") " pod="openshift-ingress/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8" Sep 07 16:30:52.214062 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:52.214018 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6235888e-5e89-4892-b007-701682139ebd-istio-data\") pod \"e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8\" (UID: \"6235888e-5e89-4892-b007-701682139ebd\") " pod="openshift-ingress/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8" Sep 07 16:30:52.214505 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:52.214484 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6235888e-5e89-4892-b007-701682139ebd-istiod-ca-cert\") pod \"e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8\" (UID: \"6235888e-5e89-4892-b007-701682139ebd\") " pod="openshift-ingress/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8" Sep 07 16:30:52.216485 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:52.216462 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6235888e-5e89-4892-b007-701682139ebd-istio-envoy\") pod \"e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8\" (UID: \"6235888e-5e89-4892-b007-701682139ebd\") " pod="openshift-ingress/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8" Sep 07 16:30:52.216635 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:52.216618 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6235888e-5e89-4892-b007-701682139ebd-istio-podinfo\") pod \"e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8\" (UID: \"6235888e-5e89-4892-b007-701682139ebd\") " pod="openshift-ingress/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8" Sep 07 16:30:52.220949 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:52.220924 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdztr\" (UniqueName: \"kubernetes.io/projected/6235888e-5e89-4892-b007-701682139ebd-kube-api-access-jdztr\") pod \"e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8\" (UID: \"6235888e-5e89-4892-b007-701682139ebd\") " pod="openshift-ingress/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8" Sep 07 16:30:52.221386 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:52.221366 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6235888e-5e89-4892-b007-701682139ebd-istio-token\") pod \"e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8\" (UID: \"6235888e-5e89-4892-b007-701682139ebd\") " pod="openshift-ingress/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8" Sep 07 16:30:52.378555 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:52.378455 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8" Sep 07 16:30:52.661122 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:52.661050 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8"] Sep 07 16:30:52.664979 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:30:52.664953 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6235888e_5e89_4892_b007_701682139ebd.slice/crio-f4a5eabb83f45cb9c684c09ea990f4baa789025190647bccfe2e9a697cf1b3d0 WatchSource:0}: Error finding container f4a5eabb83f45cb9c684c09ea990f4baa789025190647bccfe2e9a697cf1b3d0: Status 404 returned error can't find the container with id f4a5eabb83f45cb9c684c09ea990f4baa789025190647bccfe2e9a697cf1b3d0 Sep 07 16:30:52.669463 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:52.667512 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:30:52.669463 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:52.667671 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:30:52.669463 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:52.667760 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:30:53.165529 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:53.165497 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv" Sep 07 16:30:53.170944 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:53.170917 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv" Sep 07 16:30:53.179866 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:53.179838 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8" event={"ID":"6235888e-5e89-4892-b007-701682139ebd","Type":"ContainerStarted","Data":"776e99a1c4818afd5cca1a8640a81b4c50cf4860282365c379f1ca9b57a225cf"} Sep 07 16:30:53.180000 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:53.179889 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8" event={"ID":"6235888e-5e89-4892-b007-701682139ebd","Type":"ContainerStarted","Data":"f4a5eabb83f45cb9c684c09ea990f4baa789025190647bccfe2e9a697cf1b3d0"} Sep 07 16:30:53.180187 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:53.180173 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv" Sep 07 16:30:53.180828 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:53.180805 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv" Sep 07 16:30:53.210209 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:53.210164 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8" podStartSLOduration=1.210149734 podStartE2EDuration="1.210149734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:30:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:30:53.208931568 +0000 UTC m=+682.244154639" watchObservedRunningTime="2026-09-07 16:30:53.210149734 +0000 UTC m=+682.245372803" Sep 07 16:30:53.379499 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:53.379453 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8" Sep 07 16:30:53.384521 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:53.384493 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8" Sep 07 16:30:54.185324 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:54.185290 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8" Sep 07 16:30:54.186394 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:54.186373 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8" Sep 07 16:30:55.931681 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:55.931641 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-8a3403-6bd57dc687-wbk58"] Sep 07 16:30:55.935116 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:55.935084 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-8a3403-6bd57dc687-wbk58" Sep 07 16:30:55.938114 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:55.938094 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-8a3403-dockercfg-b644f\"" Sep 07 16:30:55.938587 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:55.938573 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-8a3403\"" Sep 07 16:30:55.945858 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:55.945832 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-8a3403-6bd57dc687-wbk58"] Sep 07 16:30:56.050013 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:56.049964 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e046aaf-52c5-453b-bafc-314c93f70f87-plugins-config-volume\") pod \"payload-processing-e2e-autores-8a3403-6bd57dc687-wbk58\" (UID: \"8e046aaf-52c5-453b-bafc-314c93f70f87\") " pod="openshift-ingress/payload-processing-e2e-autores-8a3403-6bd57dc687-wbk58" Sep 07 16:30:56.050208 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:56.050052 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6mh6\" (UniqueName: \"kubernetes.io/projected/8e046aaf-52c5-453b-bafc-314c93f70f87-kube-api-access-f6mh6\") pod \"payload-processing-e2e-autores-8a3403-6bd57dc687-wbk58\" (UID: \"8e046aaf-52c5-453b-bafc-314c93f70f87\") " pod="openshift-ingress/payload-processing-e2e-autores-8a3403-6bd57dc687-wbk58" Sep 07 16:30:56.151273 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:56.151232 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e046aaf-52c5-453b-bafc-314c93f70f87-plugins-config-volume\") pod \"payload-processing-e2e-autores-8a3403-6bd57dc687-wbk58\" (UID: \"8e046aaf-52c5-453b-bafc-314c93f70f87\") " pod="openshift-ingress/payload-processing-e2e-autores-8a3403-6bd57dc687-wbk58" Sep 07 16:30:56.151457 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:56.151295 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6mh6\" (UniqueName: \"kubernetes.io/projected/8e046aaf-52c5-453b-bafc-314c93f70f87-kube-api-access-f6mh6\") pod \"payload-processing-e2e-autores-8a3403-6bd57dc687-wbk58\" (UID: \"8e046aaf-52c5-453b-bafc-314c93f70f87\") " pod="openshift-ingress/payload-processing-e2e-autores-8a3403-6bd57dc687-wbk58" Sep 07 16:30:56.151998 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:56.151972 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e046aaf-52c5-453b-bafc-314c93f70f87-plugins-config-volume\") pod \"payload-processing-e2e-autores-8a3403-6bd57dc687-wbk58\" (UID: \"8e046aaf-52c5-453b-bafc-314c93f70f87\") " pod="openshift-ingress/payload-processing-e2e-autores-8a3403-6bd57dc687-wbk58" Sep 07 16:30:56.163937 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:56.163909 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6mh6\" (UniqueName: \"kubernetes.io/projected/8e046aaf-52c5-453b-bafc-314c93f70f87-kube-api-access-f6mh6\") pod \"payload-processing-e2e-autores-8a3403-6bd57dc687-wbk58\" (UID: \"8e046aaf-52c5-453b-bafc-314c93f70f87\") " pod="openshift-ingress/payload-processing-e2e-autores-8a3403-6bd57dc687-wbk58" Sep 07 16:30:56.245677 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:56.245636 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-8a3403-6bd57dc687-wbk58" Sep 07 16:30:56.532873 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:56.532849 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-8a3403-6bd57dc687-wbk58"] Sep 07 16:30:56.535359 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:30:56.535323 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e046aaf_52c5_453b_bafc_314c93f70f87.slice/crio-8ac910f7072a13e59b4dcc37a9b066b9718f679dd752ba9c27f3ef5434282857 WatchSource:0}: Error finding container 8ac910f7072a13e59b4dcc37a9b066b9718f679dd752ba9c27f3ef5434282857: Status 404 returned error can't find the container with id 8ac910f7072a13e59b4dcc37a9b066b9718f679dd752ba9c27f3ef5434282857 Sep 07 16:30:57.201056 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:57.201005 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-8a3403-6bd57dc687-wbk58" event={"ID":"8e046aaf-52c5-453b-bafc-314c93f70f87","Type":"ContainerStarted","Data":"4b02a669c1c4e252daaaedcc166dba7c0670940c01b60a79d4c8cd652b9f8e72"} Sep 07 16:30:57.201509 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:57.201079 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-8a3403-6bd57dc687-wbk58" event={"ID":"8e046aaf-52c5-453b-bafc-314c93f70f87","Type":"ContainerStarted","Data":"8ac910f7072a13e59b4dcc37a9b066b9718f679dd752ba9c27f3ef5434282857"} Sep 07 16:30:57.201509 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:57.201168 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-8a3403-6bd57dc687-wbk58" Sep 07 16:30:57.216062 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:30:57.215978 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-8a3403-6bd57dc687-wbk58" podStartSLOduration=2.215956623 podStartE2EDuration="2.215956623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:30:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:30:57.215297788 +0000 UTC m=+686.250520882" watchObservedRunningTime="2026-09-07 16:30:57.215956623 +0000 UTC m=+686.251179694" Sep 07 16:31:03.181908 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:03.181879 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-747d6d-8d79cd766-hnpxc" Sep 07 16:31:08.207076 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:08.207042 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-8a3403-6bd57dc687-wbk58" Sep 07 16:31:09.221278 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:09.221225 2580 patch_prober.go:28] interesting pod/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 07 16:31:09.221778 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:09.221310 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv" podUID="69c3fe33-98c9-404c-bccf-548c1f77e9ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 07 16:31:10.277977 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:10.277943 2580 generic.go:356] "Generic (PLEG): container finished" podID="69c3fe33-98c9-404c-bccf-548c1f77e9ec" containerID="72fe1639848a9b30d82f8423089b7e990726c8920cef18a6690012cd9de3a842" exitCode=0 Sep 07 16:31:10.278428 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:10.278017 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv" event={"ID":"69c3fe33-98c9-404c-bccf-548c1f77e9ec","Type":"ContainerDied","Data":"72fe1639848a9b30d82f8423089b7e990726c8920cef18a6690012cd9de3a842"} Sep 07 16:31:10.278501 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:10.278482 2580 scope.go:117] "RemoveContainer" containerID="72fe1639848a9b30d82f8423089b7e990726c8920cef18a6690012cd9de3a842" Sep 07 16:31:10.279580 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:10.279554 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:31:10.279652 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:10.279640 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:31:10.279687 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:10.279675 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:31:11.285771 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:11.285738 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv" event={"ID":"69c3fe33-98c9-404c-bccf-548c1f77e9ec","Type":"ContainerStarted","Data":"34f06df237fd476ee6c93a5d35feb5c0ca0d03006cf80c003e5065ecba2091c9"} Sep 07 16:31:12.165092 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:12.165052 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv" Sep 07 16:31:12.165379 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:12.165356 2580 patch_prober.go:28] interesting pod/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body= Sep 07 16:31:12.165480 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:12.165413 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv" podUID="69c3fe33-98c9-404c-bccf-548c1f77e9ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" Sep 07 16:31:13.165918 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:13.165870 2580 patch_prober.go:28] interesting pod/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body= Sep 07 16:31:13.166359 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:13.165940 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv" podUID="69c3fe33-98c9-404c-bccf-548c1f77e9ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" Sep 07 16:31:14.165248 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:14.165204 2580 patch_prober.go:28] interesting pod/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body= Sep 07 16:31:14.165451 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:14.165282 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv" podUID="69c3fe33-98c9-404c-bccf-548c1f77e9ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" Sep 07 16:31:15.165237 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:15.165135 2580 patch_prober.go:28] interesting pod/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body= Sep 07 16:31:15.165237 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:15.165192 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv" podUID="69c3fe33-98c9-404c-bccf-548c1f77e9ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" Sep 07 16:31:15.309506 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:15.309473 2580 generic.go:356] "Generic (PLEG): container finished" podID="6235888e-5e89-4892-b007-701682139ebd" containerID="776e99a1c4818afd5cca1a8640a81b4c50cf4860282365c379f1ca9b57a225cf" exitCode=0 Sep 07 16:31:15.309669 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:15.309526 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8" event={"ID":"6235888e-5e89-4892-b007-701682139ebd","Type":"ContainerDied","Data":"776e99a1c4818afd5cca1a8640a81b4c50cf4860282365c379f1ca9b57a225cf"} Sep 07 16:31:15.309905 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:15.309891 2580 scope.go:117] "RemoveContainer" containerID="776e99a1c4818afd5cca1a8640a81b4c50cf4860282365c379f1ca9b57a225cf" Sep 07 16:31:15.312120 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:15.312078 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:31:15.312236 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:15.312181 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:31:15.312296 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:15.312233 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:31:16.165063 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:16.165006 2580 patch_prober.go:28] interesting pod/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body= Sep 07 16:31:16.165276 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:16.165081 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv" podUID="69c3fe33-98c9-404c-bccf-548c1f77e9ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" Sep 07 16:31:16.318685 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:16.318648 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8" event={"ID":"6235888e-5e89-4892-b007-701682139ebd","Type":"ContainerStarted","Data":"6dcdd867766c49a143af136f5b436195562914b13528eef3f842288c765f8a43"} Sep 07 16:31:16.379223 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:16.379188 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8" Sep 07 16:31:16.379439 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:16.379415 2580 patch_prober.go:28] interesting pod/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Sep 07 16:31:16.379490 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:16.379468 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8" podUID="6235888e-5e89-4892-b007-701682139ebd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Sep 07 16:31:17.168992 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:17.168960 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv" Sep 07 16:31:17.169398 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:17.169235 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv" Sep 07 16:31:17.170037 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:17.170001 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv" Sep 07 16:31:17.379891 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:17.379852 2580 patch_prober.go:28] interesting pod/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Sep 07 16:31:17.380093 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:17.379910 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8" podUID="6235888e-5e89-4892-b007-701682139ebd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Sep 07 16:31:18.382753 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:18.382709 2580 patch_prober.go:28] interesting pod/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Sep 07 16:31:18.383184 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:18.382778 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8" podUID="6235888e-5e89-4892-b007-701682139ebd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Sep 07 16:31:19.379108 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:19.379061 2580 patch_prober.go:28] interesting pod/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Sep 07 16:31:19.379309 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:19.379136 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8" podUID="6235888e-5e89-4892-b007-701682139ebd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Sep 07 16:31:19.991017 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:19.990975 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-983d9f00-wfp69"] Sep 07 16:31:19.993960 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:19.993940 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-983d9f00-wfp69" Sep 07 16:31:19.995978 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:19.995945 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jpjxt\"" Sep 07 16:31:19.996180 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:19.996161 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 07 16:31:20.004068 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:20.004040 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-983d9f00-wfp69"] Sep 07 16:31:20.108219 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:20.108173 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65cf702b-4129-4878-93eb-94e0b595b4eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-983d9f00-wfp69\" (UID: \"65cf702b-4129-4878-93eb-94e0b595b4eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-983d9f00-wfp69" Sep 07 16:31:20.209683 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:20.209646 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65cf702b-4129-4878-93eb-94e0b595b4eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-983d9f00-wfp69\" (UID: \"65cf702b-4129-4878-93eb-94e0b595b4eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-983d9f00-wfp69" Sep 07 16:31:20.210630 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:20.210599 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65cf702b-4129-4878-93eb-94e0b595b4eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-983d9f00-wfp69\" (UID: \"65cf702b-4129-4878-93eb-94e0b595b4eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-983d9f00-wfp69" Sep 07 16:31:20.304388 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:20.304292 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-983d9f00-wfp69" Sep 07 16:31:20.380189 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:20.379673 2580 patch_prober.go:28] interesting pod/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Sep 07 16:31:20.380189 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:20.379752 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8" podUID="6235888e-5e89-4892-b007-701682139ebd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Sep 07 16:31:20.590605 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:20.590576 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-983d9f00-wfp69"] Sep 07 16:31:20.592869 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:31:20.592839 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65cf702b_4129_4878_93eb_94e0b595b4eb.slice/crio-fbbbde3de9a5bd13f27d40dcb3e0d9b496642d9194d6504da30976fc1b3b0eda WatchSource:0}: Error finding container fbbbde3de9a5bd13f27d40dcb3e0d9b496642d9194d6504da30976fc1b3b0eda: Status 404 returned error can't find the container with id fbbbde3de9a5bd13f27d40dcb3e0d9b496642d9194d6504da30976fc1b3b0eda Sep 07 16:31:21.354815 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:21.354777 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-983d9f00-wfp69" event={"ID":"65cf702b-4129-4878-93eb-94e0b595b4eb","Type":"ContainerStarted","Data":"fbbbde3de9a5bd13f27d40dcb3e0d9b496642d9194d6504da30976fc1b3b0eda"} Sep 07 16:31:21.379200 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:21.379164 2580 patch_prober.go:28] interesting pod/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Sep 07 16:31:21.379329 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:21.379226 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8" podUID="6235888e-5e89-4892-b007-701682139ebd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Sep 07 16:31:22.383493 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:22.383399 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8" Sep 07 16:31:22.384053 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:22.383762 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8" Sep 07 16:31:22.384861 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:22.384836 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8" Sep 07 16:31:23.365938 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:23.365842 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-983d9f00-wfp69" event={"ID":"65cf702b-4129-4878-93eb-94e0b595b4eb","Type":"ContainerStarted","Data":"d31551bdd53f454cc6a17cebe9bbfef8611a149de37a9a7dab2dabc26fb2709e"} Sep 07 16:31:23.382376 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:23.382318 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-983d9f00-wfp69" podStartSLOduration=1.982225218 podStartE2EDuration="4.382300043s" totalImagesPullingTime="2.400074825s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:31:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:31:23.380492822 +0000 UTC m=+712.415715892" watchObservedRunningTime="2026-09-07 16:31:23.382300043 +0000 UTC m=+712.417523143" Sep 07 16:31:24.373360 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:24.373324 2580 generic.go:356] "Generic (PLEG): container finished" podID="65cf702b-4129-4878-93eb-94e0b595b4eb" containerID="d31551bdd53f454cc6a17cebe9bbfef8611a149de37a9a7dab2dabc26fb2709e" exitCode=0 Sep 07 16:31:24.373823 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:24.373412 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-983d9f00-wfp69" event={"ID":"65cf702b-4129-4878-93eb-94e0b595b4eb","Type":"ContainerDied","Data":"d31551bdd53f454cc6a17cebe9bbfef8611a149de37a9a7dab2dabc26fb2709e"} Sep 07 16:31:25.509202 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:25.509177 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-983d9f00-wfp69" Sep 07 16:31:25.672995 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:25.672910 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65cf702b-4129-4878-93eb-94e0b595b4eb-maas-api-service-ca\") pod \"65cf702b-4129-4878-93eb-94e0b595b4eb\" (UID: \"65cf702b-4129-4878-93eb-94e0b595b4eb\") " Sep 07 16:31:25.673289 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:25.673265 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65cf702b-4129-4878-93eb-94e0b595b4eb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "65cf702b-4129-4878-93eb-94e0b595b4eb" (UID: "65cf702b-4129-4878-93eb-94e0b595b4eb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 16:31:25.773872 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:25.773825 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65cf702b-4129-4878-93eb-94e0b595b4eb-maas-api-service-ca\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:31:26.384062 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:26.384011 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-983d9f00-wfp69" event={"ID":"65cf702b-4129-4878-93eb-94e0b595b4eb","Type":"ContainerDied","Data":"fbbbde3de9a5bd13f27d40dcb3e0d9b496642d9194d6504da30976fc1b3b0eda"} Sep 07 16:31:26.384062 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:26.384064 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-983d9f00-wfp69" Sep 07 16:31:26.384266 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:26.384078 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fbbbde3de9a5bd13f27d40dcb3e0d9b496642d9194d6504da30976fc1b3b0eda" Sep 07 16:31:36.372786 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:36.372750 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-983d9f00-wfp69"] Sep 07 16:31:36.376206 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:36.376178 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-983d9f00-wfp69"] Sep 07 16:31:36.486273 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:36.486242 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8"] Sep 07 16:31:36.486586 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:36.486558 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8" podUID="6235888e-5e89-4892-b007-701682139ebd" containerName="istio-proxy" containerID="cri-o://6dcdd867766c49a143af136f5b436195562914b13528eef3f842288c765f8a43" gracePeriod=30 Sep 07 16:31:37.386704 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:37.386657 2580 patch_prober.go:28] interesting pod/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 07 16:31:37.387122 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:37.386715 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8" podUID="6235888e-5e89-4892-b007-701682139ebd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 07 16:31:37.422139 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:37.422102 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65cf702b-4129-4878-93eb-94e0b595b4eb" path="/var/lib/kubelet/pods/65cf702b-4129-4878-93eb-94e0b595b4eb/volumes" Sep 07 16:31:41.686597 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:41.686563 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-1e4cf2b6-84cc97bbdb-h5sjl"] Sep 07 16:31:41.687174 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:41.687154 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65cf702b-4129-4878-93eb-94e0b595b4eb" containerName="revoke-keys" Sep 07 16:31:41.687222 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:41.687180 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="65cf702b-4129-4878-93eb-94e0b595b4eb" containerName="revoke-keys" Sep 07 16:31:41.687284 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:41.687272 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="65cf702b-4129-4878-93eb-94e0b595b4eb" containerName="revoke-keys" Sep 07 16:31:41.689581 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:41.689563 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1e4cf2b6-84cc97bbdb-h5sjl" Sep 07 16:31:41.692046 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:41.691993 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-1e4cf2b6\"" Sep 07 16:31:41.713406 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:41.713368 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-1e4cf2b6-84cc97bbdb-h5sjl"] Sep 07 16:31:41.750511 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:41.750444 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1e4cf2b6-f99494db6-cnw8z"] Sep 07 16:31:41.752743 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:41.752720 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-1e4cf2b6-f99494db6-cnw8z" Sep 07 16:31:41.755224 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:41.755208 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-1e4cf2b6-dockercfg-9zpps\"" Sep 07 16:31:41.758091 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:41.758072 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8" Sep 07 16:31:41.773476 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:41.773446 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1e4cf2b6-f99494db6-cnw8z"] Sep 07 16:31:41.825539 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:41.822408 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6235888e-5e89-4892-b007-701682139ebd-istio-podinfo\") pod \"6235888e-5e89-4892-b007-701682139ebd\" (UID: \"6235888e-5e89-4892-b007-701682139ebd\") " Sep 07 16:31:41.825539 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:41.822459 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6235888e-5e89-4892-b007-701682139ebd-istio-envoy\") pod \"6235888e-5e89-4892-b007-701682139ebd\" (UID: \"6235888e-5e89-4892-b007-701682139ebd\") " Sep 07 16:31:41.825539 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:41.822504 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6235888e-5e89-4892-b007-701682139ebd-istiod-ca-cert\") pod \"6235888e-5e89-4892-b007-701682139ebd\" (UID: \"6235888e-5e89-4892-b007-701682139ebd\") " Sep 07 16:31:41.825539 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:41.822531 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jdztr\" (UniqueName: \"kubernetes.io/projected/6235888e-5e89-4892-b007-701682139ebd-kube-api-access-jdztr\") pod \"6235888e-5e89-4892-b007-701682139ebd\" (UID: \"6235888e-5e89-4892-b007-701682139ebd\") " Sep 07 16:31:41.825539 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:41.822566 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6235888e-5e89-4892-b007-701682139ebd-credential-socket\") pod \"6235888e-5e89-4892-b007-701682139ebd\" (UID: \"6235888e-5e89-4892-b007-701682139ebd\") " Sep 07 16:31:41.825539 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:41.823385 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6235888e-5e89-4892-b007-701682139ebd-istio-token\") pod \"6235888e-5e89-4892-b007-701682139ebd\" (UID: \"6235888e-5e89-4892-b007-701682139ebd\") " Sep 07 16:31:41.825539 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:41.823445 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6235888e-5e89-4892-b007-701682139ebd-workload-certs\") pod \"6235888e-5e89-4892-b007-701682139ebd\" (UID: \"6235888e-5e89-4892-b007-701682139ebd\") " Sep 07 16:31:41.825539 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:41.823488 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6235888e-5e89-4892-b007-701682139ebd-istio-data\") pod \"6235888e-5e89-4892-b007-701682139ebd\" (UID: \"6235888e-5e89-4892-b007-701682139ebd\") " Sep 07 16:31:41.825539 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:41.823533 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6235888e-5e89-4892-b007-701682139ebd-workload-socket\") pod \"6235888e-5e89-4892-b007-701682139ebd\" (UID: \"6235888e-5e89-4892-b007-701682139ebd\") " Sep 07 16:31:41.825539 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:41.823687 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pmjx\" (UniqueName: \"kubernetes.io/projected/84d7d6ff-755a-4145-8c74-23c6e4af6332-kube-api-access-2pmjx\") pod \"payload-processing-e2e-migrate-1e4cf2b6-f99494db6-cnw8z\" (UID: \"84d7d6ff-755a-4145-8c74-23c6e4af6332\") " pod="openshift-ingress/payload-processing-e2e-migrate-1e4cf2b6-f99494db6-cnw8z" Sep 07 16:31:41.825539 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:41.823811 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84d7d6ff-755a-4145-8c74-23c6e4af6332-plugins-config-volume\") pod \"payload-processing-e2e-migrate-1e4cf2b6-f99494db6-cnw8z\" (UID: \"84d7d6ff-755a-4145-8c74-23c6e4af6332\") " pod="openshift-ingress/payload-processing-e2e-migrate-1e4cf2b6-f99494db6-cnw8z" Sep 07 16:31:41.825539 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:41.823865 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3aeff7e5-c168-4f71-846b-eae0ed630bdf-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-1e4cf2b6-84cc97bbdb-h5sjl\" (UID: \"3aeff7e5-c168-4f71-846b-eae0ed630bdf\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-1e4cf2b6-84cc97bbdb-h5sjl" Sep 07 16:31:41.825539 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:41.823917 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76sgh\" (UniqueName: \"kubernetes.io/projected/3aeff7e5-c168-4f71-846b-eae0ed630bdf-kube-api-access-76sgh\") pod \"payload-pre-processing-e2e-migrate-1e4cf2b6-84cc97bbdb-h5sjl\" (UID: \"3aeff7e5-c168-4f71-846b-eae0ed630bdf\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-1e4cf2b6-84cc97bbdb-h5sjl" Sep 07 16:31:41.825539 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:41.824394 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6235888e-5e89-4892-b007-701682139ebd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6235888e-5e89-4892-b007-701682139ebd" (UID: "6235888e-5e89-4892-b007-701682139ebd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 16:31:41.825539 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:41.824909 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6235888e-5e89-4892-b007-701682139ebd-istio-data" (OuterVolumeSpecName: "istio-data") pod "6235888e-5e89-4892-b007-701682139ebd" (UID: "6235888e-5e89-4892-b007-701682139ebd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 16:31:41.825539 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:41.825116 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6235888e-5e89-4892-b007-701682139ebd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6235888e-5e89-4892-b007-701682139ebd" (UID: "6235888e-5e89-4892-b007-701682139ebd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 16:31:41.826938 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:41.825260 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6235888e-5e89-4892-b007-701682139ebd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6235888e-5e89-4892-b007-701682139ebd" (UID: "6235888e-5e89-4892-b007-701682139ebd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 16:31:41.826938 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:41.825480 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6235888e-5e89-4892-b007-701682139ebd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6235888e-5e89-4892-b007-701682139ebd" (UID: "6235888e-5e89-4892-b007-701682139ebd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 16:31:41.827156 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:41.827137 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6235888e-5e89-4892-b007-701682139ebd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6235888e-5e89-4892-b007-701682139ebd" (UID: "6235888e-5e89-4892-b007-701682139ebd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 16:31:41.828404 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:41.828368 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6235888e-5e89-4892-b007-701682139ebd-kube-api-access-jdztr" (OuterVolumeSpecName: "kube-api-access-jdztr") pod "6235888e-5e89-4892-b007-701682139ebd" (UID: "6235888e-5e89-4892-b007-701682139ebd"). InnerVolumeSpecName "kube-api-access-jdztr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 16:31:41.828506 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:41.828399 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6235888e-5e89-4892-b007-701682139ebd-istio-token" (OuterVolumeSpecName: "istio-token") pod "6235888e-5e89-4892-b007-701682139ebd" (UID: "6235888e-5e89-4892-b007-701682139ebd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 16:31:41.828762 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:41.828638 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6235888e-5e89-4892-b007-701682139ebd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6235888e-5e89-4892-b007-701682139ebd" (UID: "6235888e-5e89-4892-b007-701682139ebd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 07 16:31:41.925265 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:41.925226 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84d7d6ff-755a-4145-8c74-23c6e4af6332-plugins-config-volume\") pod \"payload-processing-e2e-migrate-1e4cf2b6-f99494db6-cnw8z\" (UID: \"84d7d6ff-755a-4145-8c74-23c6e4af6332\") " pod="openshift-ingress/payload-processing-e2e-migrate-1e4cf2b6-f99494db6-cnw8z" Sep 07 16:31:41.925457 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:41.925282 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3aeff7e5-c168-4f71-846b-eae0ed630bdf-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-1e4cf2b6-84cc97bbdb-h5sjl\" (UID: \"3aeff7e5-c168-4f71-846b-eae0ed630bdf\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-1e4cf2b6-84cc97bbdb-h5sjl" Sep 07 16:31:41.925457 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:41.925317 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76sgh\" (UniqueName: \"kubernetes.io/projected/3aeff7e5-c168-4f71-846b-eae0ed630bdf-kube-api-access-76sgh\") pod \"payload-pre-processing-e2e-migrate-1e4cf2b6-84cc97bbdb-h5sjl\" (UID: \"3aeff7e5-c168-4f71-846b-eae0ed630bdf\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-1e4cf2b6-84cc97bbdb-h5sjl" Sep 07 16:31:41.925457 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:41.925353 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pmjx\" (UniqueName: \"kubernetes.io/projected/84d7d6ff-755a-4145-8c74-23c6e4af6332-kube-api-access-2pmjx\") pod \"payload-processing-e2e-migrate-1e4cf2b6-f99494db6-cnw8z\" (UID: \"84d7d6ff-755a-4145-8c74-23c6e4af6332\") " pod="openshift-ingress/payload-processing-e2e-migrate-1e4cf2b6-f99494db6-cnw8z" Sep 07 16:31:41.925457 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:41.925417 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6235888e-5e89-4892-b007-701682139ebd-istiod-ca-cert\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:31:41.925457 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:41.925431 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdztr\" (UniqueName: \"kubernetes.io/projected/6235888e-5e89-4892-b007-701682139ebd-kube-api-access-jdztr\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:31:41.925457 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:41.925447 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6235888e-5e89-4892-b007-701682139ebd-credential-socket\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:31:41.925457 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:41.925460 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6235888e-5e89-4892-b007-701682139ebd-istio-token\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:31:41.925828 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:41.925472 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6235888e-5e89-4892-b007-701682139ebd-workload-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:31:41.925828 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:41.925485 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6235888e-5e89-4892-b007-701682139ebd-istio-data\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:31:41.925828 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:41.925497 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6235888e-5e89-4892-b007-701682139ebd-workload-socket\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:31:41.925828 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:41.925508 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6235888e-5e89-4892-b007-701682139ebd-istio-podinfo\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:31:41.925828 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:41.925520 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6235888e-5e89-4892-b007-701682139ebd-istio-envoy\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:31:41.925972 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:41.925914 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84d7d6ff-755a-4145-8c74-23c6e4af6332-plugins-config-volume\") pod \"payload-processing-e2e-migrate-1e4cf2b6-f99494db6-cnw8z\" (UID: \"84d7d6ff-755a-4145-8c74-23c6e4af6332\") " pod="openshift-ingress/payload-processing-e2e-migrate-1e4cf2b6-f99494db6-cnw8z" Sep 07 16:31:41.926005 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:41.925988 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3aeff7e5-c168-4f71-846b-eae0ed630bdf-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-1e4cf2b6-84cc97bbdb-h5sjl\" (UID: \"3aeff7e5-c168-4f71-846b-eae0ed630bdf\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-1e4cf2b6-84cc97bbdb-h5sjl" Sep 07 16:31:41.932858 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:41.932826 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76sgh\" (UniqueName: \"kubernetes.io/projected/3aeff7e5-c168-4f71-846b-eae0ed630bdf-kube-api-access-76sgh\") pod \"payload-pre-processing-e2e-migrate-1e4cf2b6-84cc97bbdb-h5sjl\" (UID: \"3aeff7e5-c168-4f71-846b-eae0ed630bdf\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-1e4cf2b6-84cc97bbdb-h5sjl" Sep 07 16:31:41.932966 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:41.932870 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pmjx\" (UniqueName: \"kubernetes.io/projected/84d7d6ff-755a-4145-8c74-23c6e4af6332-kube-api-access-2pmjx\") pod \"payload-processing-e2e-migrate-1e4cf2b6-f99494db6-cnw8z\" (UID: \"84d7d6ff-755a-4145-8c74-23c6e4af6332\") " pod="openshift-ingress/payload-processing-e2e-migrate-1e4cf2b6-f99494db6-cnw8z" Sep 07 16:31:41.999963 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:41.999931 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1e4cf2b6-84cc97bbdb-h5sjl" Sep 07 16:31:42.063592 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:42.063417 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-1e4cf2b6-f99494db6-cnw8z" Sep 07 16:31:42.273452 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:42.273427 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-1e4cf2b6-84cc97bbdb-h5sjl"] Sep 07 16:31:42.275946 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:31:42.275920 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aeff7e5_c168_4f71_846b_eae0ed630bdf.slice/crio-f54e82e05a94a054e15765eec06538c4feda2679f4964c2ce23d30da6af6d67b WatchSource:0}: Error finding container f54e82e05a94a054e15765eec06538c4feda2679f4964c2ce23d30da6af6d67b: Status 404 returned error can't find the container with id f54e82e05a94a054e15765eec06538c4feda2679f4964c2ce23d30da6af6d67b Sep 07 16:31:42.340119 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:42.340085 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1e4cf2b6-f99494db6-cnw8z"] Sep 07 16:31:42.343637 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:31:42.343612 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84d7d6ff_755a_4145_8c74_23c6e4af6332.slice/crio-85400a00845d7a2cb9524e9c7b9c6e3a02bd07613fbc3757bb5ec7d1c88f8cd9 WatchSource:0}: Error finding container 85400a00845d7a2cb9524e9c7b9c6e3a02bd07613fbc3757bb5ec7d1c88f8cd9: Status 404 returned error can't find the container with id 85400a00845d7a2cb9524e9c7b9c6e3a02bd07613fbc3757bb5ec7d1c88f8cd9 Sep 07 16:31:42.458170 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:42.458129 2580 generic.go:356] "Generic (PLEG): container finished" podID="6235888e-5e89-4892-b007-701682139ebd" containerID="6dcdd867766c49a143af136f5b436195562914b13528eef3f842288c765f8a43" exitCode=0 Sep 07 16:31:42.458379 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:42.458228 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8" event={"ID":"6235888e-5e89-4892-b007-701682139ebd","Type":"ContainerDied","Data":"6dcdd867766c49a143af136f5b436195562914b13528eef3f842288c765f8a43"} Sep 07 16:31:42.458379 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:42.458251 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8" Sep 07 16:31:42.458379 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:42.458290 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8" event={"ID":"6235888e-5e89-4892-b007-701682139ebd","Type":"ContainerDied","Data":"f4a5eabb83f45cb9c684c09ea990f4baa789025190647bccfe2e9a697cf1b3d0"} Sep 07 16:31:42.458379 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:42.458329 2580 scope.go:117] "RemoveContainer" containerID="6dcdd867766c49a143af136f5b436195562914b13528eef3f842288c765f8a43" Sep 07 16:31:42.460463 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:42.460428 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-1e4cf2b6-f99494db6-cnw8z" event={"ID":"84d7d6ff-755a-4145-8c74-23c6e4af6332","Type":"ContainerStarted","Data":"7eda34e7bf31a9801b23c5e8433c8764213f5ad304fd5a3d9544a81dad2fe518"} Sep 07 16:31:42.460583 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:42.460488 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-1e4cf2b6-f99494db6-cnw8z" event={"ID":"84d7d6ff-755a-4145-8c74-23c6e4af6332","Type":"ContainerStarted","Data":"85400a00845d7a2cb9524e9c7b9c6e3a02bd07613fbc3757bb5ec7d1c88f8cd9"} Sep 07 16:31:42.460583 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:42.460527 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-1e4cf2b6-f99494db6-cnw8z" Sep 07 16:31:42.463231 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:42.463199 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1e4cf2b6-84cc97bbdb-h5sjl" event={"ID":"3aeff7e5-c168-4f71-846b-eae0ed630bdf","Type":"ContainerStarted","Data":"6990ae8bdacb3a763e538dc2a08cd0b046bfa271f67476be05cca0af09d7adfa"} Sep 07 16:31:42.463340 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:42.463260 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1e4cf2b6-84cc97bbdb-h5sjl" event={"ID":"3aeff7e5-c168-4f71-846b-eae0ed630bdf","Type":"ContainerStarted","Data":"f54e82e05a94a054e15765eec06538c4feda2679f4964c2ce23d30da6af6d67b"} Sep 07 16:31:42.463532 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:42.463504 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1e4cf2b6-84cc97bbdb-h5sjl" Sep 07 16:31:42.469317 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:42.469300 2580 scope.go:117] "RemoveContainer" containerID="776e99a1c4818afd5cca1a8640a81b4c50cf4860282365c379f1ca9b57a225cf" Sep 07 16:31:42.476161 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:42.475560 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-1e4cf2b6-f99494db6-cnw8z" podStartSLOduration=1.47554119 podStartE2EDuration="1.47554119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:31:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:31:42.475191099 +0000 UTC m=+731.510414172" watchObservedRunningTime="2026-09-07 16:31:42.47554119 +0000 UTC m=+731.510764261" Sep 07 16:31:42.478428 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:42.478412 2580 scope.go:117] "RemoveContainer" containerID="6dcdd867766c49a143af136f5b436195562914b13528eef3f842288c765f8a43" Sep 07 16:31:42.478754 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:31:42.478733 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dcdd867766c49a143af136f5b436195562914b13528eef3f842288c765f8a43\": container with ID starting with 6dcdd867766c49a143af136f5b436195562914b13528eef3f842288c765f8a43 not found: ID does not exist" containerID="6dcdd867766c49a143af136f5b436195562914b13528eef3f842288c765f8a43" Sep 07 16:31:42.478796 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:42.478763 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dcdd867766c49a143af136f5b436195562914b13528eef3f842288c765f8a43"} err="failed to get container status \"6dcdd867766c49a143af136f5b436195562914b13528eef3f842288c765f8a43\": rpc error: code = NotFound desc = could not find container \"6dcdd867766c49a143af136f5b436195562914b13528eef3f842288c765f8a43\": container with ID starting with 6dcdd867766c49a143af136f5b436195562914b13528eef3f842288c765f8a43 not found: ID does not exist" Sep 07 16:31:42.478796 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:42.478781 2580 scope.go:117] "RemoveContainer" containerID="776e99a1c4818afd5cca1a8640a81b4c50cf4860282365c379f1ca9b57a225cf" Sep 07 16:31:42.479232 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:31:42.479015 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"776e99a1c4818afd5cca1a8640a81b4c50cf4860282365c379f1ca9b57a225cf\": container with ID starting with 776e99a1c4818afd5cca1a8640a81b4c50cf4860282365c379f1ca9b57a225cf not found: ID does not exist" containerID="776e99a1c4818afd5cca1a8640a81b4c50cf4860282365c379f1ca9b57a225cf" Sep 07 16:31:42.479232 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:42.479048 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"776e99a1c4818afd5cca1a8640a81b4c50cf4860282365c379f1ca9b57a225cf"} err="failed to get container status \"776e99a1c4818afd5cca1a8640a81b4c50cf4860282365c379f1ca9b57a225cf\": rpc error: code = NotFound desc = could not find container \"776e99a1c4818afd5cca1a8640a81b4c50cf4860282365c379f1ca9b57a225cf\": container with ID starting with 776e99a1c4818afd5cca1a8640a81b4c50cf4860282365c379f1ca9b57a225cf not found: ID does not exist" Sep 07 16:31:42.490698 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:42.490659 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1e4cf2b6-84cc97bbdb-h5sjl" podStartSLOduration=1.490646671 podStartE2EDuration="1.490646671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:31:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:31:42.488950681 +0000 UTC m=+731.524173751" watchObservedRunningTime="2026-09-07 16:31:42.490646671 +0000 UTC m=+731.525869754" Sep 07 16:31:42.505404 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:42.505380 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8"] Sep 07 16:31:42.509006 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:42.508986 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-983d9f00-openshift-default-76f6d94c74-k22z8"] Sep 07 16:31:43.418379 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:43.418330 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6235888e-5e89-4892-b007-701682139ebd" path="/var/lib/kubelet/pods/6235888e-5e89-4892-b007-701682139ebd/volumes" Sep 07 16:31:53.470396 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:53.470349 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-1e4cf2b6-f99494db6-cnw8z" Sep 07 16:31:53.470885 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:31:53.470436 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1e4cf2b6-84cc97bbdb-h5sjl" Sep 07 16:32:05.387677 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:05.387636 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1e4cf2b6-mgzjf"] Sep 07 16:32:05.388082 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:05.387979 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6235888e-5e89-4892-b007-701682139ebd" containerName="istio-proxy" Sep 07 16:32:05.388082 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:05.387991 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6235888e-5e89-4892-b007-701682139ebd" containerName="istio-proxy" Sep 07 16:32:05.388082 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:05.387999 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6235888e-5e89-4892-b007-701682139ebd" containerName="istio-proxy" Sep 07 16:32:05.388082 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:05.388005 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6235888e-5e89-4892-b007-701682139ebd" containerName="istio-proxy" Sep 07 16:32:05.388214 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:05.388086 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6235888e-5e89-4892-b007-701682139ebd" containerName="istio-proxy" Sep 07 16:32:05.391093 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:05.391075 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1e4cf2b6-mgzjf" Sep 07 16:32:05.393527 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:05.393500 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 07 16:32:05.393678 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:05.393500 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jpjxt\"" Sep 07 16:32:05.398015 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:05.397987 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1e4cf2b6-mgzjf"] Sep 07 16:32:05.546542 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:05.546491 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3624b12-0150-4711-8863-33bb7104662a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1e4cf2b6-mgzjf\" (UID: \"f3624b12-0150-4711-8863-33bb7104662a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1e4cf2b6-mgzjf" Sep 07 16:32:05.647964 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:05.647861 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3624b12-0150-4711-8863-33bb7104662a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1e4cf2b6-mgzjf\" (UID: \"f3624b12-0150-4711-8863-33bb7104662a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1e4cf2b6-mgzjf" Sep 07 16:32:05.648547 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:05.648523 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3624b12-0150-4711-8863-33bb7104662a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1e4cf2b6-mgzjf\" (UID: \"f3624b12-0150-4711-8863-33bb7104662a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1e4cf2b6-mgzjf" Sep 07 16:32:05.703922 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:05.703882 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1e4cf2b6-mgzjf" Sep 07 16:32:05.981788 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:05.981754 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1e4cf2b6-mgzjf"] Sep 07 16:32:05.985263 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:32:05.985232 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3624b12_0150_4711_8863_33bb7104662a.slice/crio-53d8a5143513ce1c7e705d566e49bd8d47385bcadb3c772767f57ff163ffffc1 WatchSource:0}: Error finding container 53d8a5143513ce1c7e705d566e49bd8d47385bcadb3c772767f57ff163ffffc1: Status 404 returned error can't find the container with id 53d8a5143513ce1c7e705d566e49bd8d47385bcadb3c772767f57ff163ffffc1 Sep 07 16:32:06.578672 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:06.578578 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1e4cf2b6-mgzjf" event={"ID":"f3624b12-0150-4711-8863-33bb7104662a","Type":"ContainerStarted","Data":"bd0802133c9fe0d9c9c58a111e03b2536999215571520ee9a32872fd6936aa39"} Sep 07 16:32:06.578672 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:06.578640 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1e4cf2b6-mgzjf" event={"ID":"f3624b12-0150-4711-8863-33bb7104662a","Type":"ContainerStarted","Data":"53d8a5143513ce1c7e705d566e49bd8d47385bcadb3c772767f57ff163ffffc1"} Sep 07 16:32:06.593540 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:06.593489 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1e4cf2b6-mgzjf" podStartSLOduration=1.5934737 podStartE2EDuration="1.5934737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:32:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:32:06.591877427 +0000 UTC m=+755.627100497" watchObservedRunningTime="2026-09-07 16:32:06.5934737 +0000 UTC m=+755.628696769" Sep 07 16:32:07.590194 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:07.590156 2580 generic.go:356] "Generic (PLEG): container finished" podID="f3624b12-0150-4711-8863-33bb7104662a" containerID="bd0802133c9fe0d9c9c58a111e03b2536999215571520ee9a32872fd6936aa39" exitCode=0 Sep 07 16:32:07.590641 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:07.590240 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1e4cf2b6-mgzjf" event={"ID":"f3624b12-0150-4711-8863-33bb7104662a","Type":"ContainerDied","Data":"bd0802133c9fe0d9c9c58a111e03b2536999215571520ee9a32872fd6936aa39"} Sep 07 16:32:08.723103 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:08.723078 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1e4cf2b6-mgzjf" Sep 07 16:32:08.771250 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:08.771217 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3624b12-0150-4711-8863-33bb7104662a-maas-api-service-ca\") pod \"f3624b12-0150-4711-8863-33bb7104662a\" (UID: \"f3624b12-0150-4711-8863-33bb7104662a\") " Sep 07 16:32:08.771576 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:08.771556 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3624b12-0150-4711-8863-33bb7104662a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f3624b12-0150-4711-8863-33bb7104662a" (UID: "f3624b12-0150-4711-8863-33bb7104662a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 16:32:08.872353 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:08.872267 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3624b12-0150-4711-8863-33bb7104662a-maas-api-service-ca\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:32:09.601603 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:09.601577 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1e4cf2b6-mgzjf" Sep 07 16:32:09.601813 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:09.601608 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1e4cf2b6-mgzjf" event={"ID":"f3624b12-0150-4711-8863-33bb7104662a","Type":"ContainerDied","Data":"53d8a5143513ce1c7e705d566e49bd8d47385bcadb3c772767f57ff163ffffc1"} Sep 07 16:32:09.601813 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:09.601654 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53d8a5143513ce1c7e705d566e49bd8d47385bcadb3c772767f57ff163ffffc1" Sep 07 16:32:11.969155 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:11.969121 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1e4cf2b6-f99494db6-cnw8z"] Sep 07 16:32:11.969531 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:11.969324 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-1e4cf2b6-f99494db6-cnw8z" podUID="84d7d6ff-755a-4145-8c74-23c6e4af6332" containerName="payload-processing" containerID="cri-o://7eda34e7bf31a9801b23c5e8433c8764213f5ad304fd5a3d9544a81dad2fe518" gracePeriod=30 Sep 07 16:32:11.980717 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:11.980685 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-1e4cf2b6-84cc97bbdb-h5sjl"] Sep 07 16:32:11.980903 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:11.980882 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1e4cf2b6-84cc97bbdb-h5sjl" podUID="3aeff7e5-c168-4f71-846b-eae0ed630bdf" containerName="payload-pre-processing" containerID="cri-o://6990ae8bdacb3a763e538dc2a08cd0b046bfa271f67476be05cca0af09d7adfa" gracePeriod=30 Sep 07 16:32:12.153049 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:12.152976 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1e4cf2b6-mgzjf"] Sep 07 16:32:12.155849 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:12.155807 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1e4cf2b6-mgzjf"] Sep 07 16:32:12.233567 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:12.233504 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-1e4cf2b6-f99494db6-cnw8z" Sep 07 16:32:12.263868 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:12.263840 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1e4cf2b6-84cc97bbdb-h5sjl" Sep 07 16:32:12.304523 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:12.304487 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-76sgh\" (UniqueName: \"kubernetes.io/projected/3aeff7e5-c168-4f71-846b-eae0ed630bdf-kube-api-access-76sgh\") pod \"3aeff7e5-c168-4f71-846b-eae0ed630bdf\" (UID: \"3aeff7e5-c168-4f71-846b-eae0ed630bdf\") " Sep 07 16:32:12.304687 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:12.304636 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3aeff7e5-c168-4f71-846b-eae0ed630bdf-plugins-config-volume\") pod \"3aeff7e5-c168-4f71-846b-eae0ed630bdf\" (UID: \"3aeff7e5-c168-4f71-846b-eae0ed630bdf\") " Sep 07 16:32:12.304687 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:12.304670 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2pmjx\" (UniqueName: \"kubernetes.io/projected/84d7d6ff-755a-4145-8c74-23c6e4af6332-kube-api-access-2pmjx\") pod \"84d7d6ff-755a-4145-8c74-23c6e4af6332\" (UID: \"84d7d6ff-755a-4145-8c74-23c6e4af6332\") " Sep 07 16:32:12.304757 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:12.304716 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84d7d6ff-755a-4145-8c74-23c6e4af6332-plugins-config-volume\") pod \"84d7d6ff-755a-4145-8c74-23c6e4af6332\" (UID: \"84d7d6ff-755a-4145-8c74-23c6e4af6332\") " Sep 07 16:32:12.305047 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:12.304979 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aeff7e5-c168-4f71-846b-eae0ed630bdf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3aeff7e5-c168-4f71-846b-eae0ed630bdf" (UID: "3aeff7e5-c168-4f71-846b-eae0ed630bdf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 16:32:12.305343 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:12.305170 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84d7d6ff-755a-4145-8c74-23c6e4af6332-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "84d7d6ff-755a-4145-8c74-23c6e4af6332" (UID: "84d7d6ff-755a-4145-8c74-23c6e4af6332"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 16:32:12.306940 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:12.306903 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aeff7e5-c168-4f71-846b-eae0ed630bdf-kube-api-access-76sgh" (OuterVolumeSpecName: "kube-api-access-76sgh") pod "3aeff7e5-c168-4f71-846b-eae0ed630bdf" (UID: "3aeff7e5-c168-4f71-846b-eae0ed630bdf"). InnerVolumeSpecName "kube-api-access-76sgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 16:32:12.307175 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:12.307146 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84d7d6ff-755a-4145-8c74-23c6e4af6332-kube-api-access-2pmjx" (OuterVolumeSpecName: "kube-api-access-2pmjx") pod "84d7d6ff-755a-4145-8c74-23c6e4af6332" (UID: "84d7d6ff-755a-4145-8c74-23c6e4af6332"). InnerVolumeSpecName "kube-api-access-2pmjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 16:32:12.410913 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:12.406010 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3aeff7e5-c168-4f71-846b-eae0ed630bdf-plugins-config-volume\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:32:12.410913 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:12.406078 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pmjx\" (UniqueName: \"kubernetes.io/projected/84d7d6ff-755a-4145-8c74-23c6e4af6332-kube-api-access-2pmjx\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:32:12.410913 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:12.406094 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84d7d6ff-755a-4145-8c74-23c6e4af6332-plugins-config-volume\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:32:12.410913 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:12.406109 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76sgh\" (UniqueName: \"kubernetes.io/projected/3aeff7e5-c168-4f71-846b-eae0ed630bdf-kube-api-access-76sgh\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:32:12.616933 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:12.616848 2580 generic.go:356] "Generic (PLEG): container finished" podID="84d7d6ff-755a-4145-8c74-23c6e4af6332" containerID="7eda34e7bf31a9801b23c5e8433c8764213f5ad304fd5a3d9544a81dad2fe518" exitCode=0 Sep 07 16:32:12.616933 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:12.616902 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-1e4cf2b6-f99494db6-cnw8z" Sep 07 16:32:12.617190 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:12.616929 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-1e4cf2b6-f99494db6-cnw8z" event={"ID":"84d7d6ff-755a-4145-8c74-23c6e4af6332","Type":"ContainerDied","Data":"7eda34e7bf31a9801b23c5e8433c8764213f5ad304fd5a3d9544a81dad2fe518"} Sep 07 16:32:12.617190 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:12.616980 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-1e4cf2b6-f99494db6-cnw8z" event={"ID":"84d7d6ff-755a-4145-8c74-23c6e4af6332","Type":"ContainerDied","Data":"85400a00845d7a2cb9524e9c7b9c6e3a02bd07613fbc3757bb5ec7d1c88f8cd9"} Sep 07 16:32:12.617190 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:12.617009 2580 scope.go:117] "RemoveContainer" containerID="7eda34e7bf31a9801b23c5e8433c8764213f5ad304fd5a3d9544a81dad2fe518" Sep 07 16:32:12.618519 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:12.618489 2580 generic.go:356] "Generic (PLEG): container finished" podID="3aeff7e5-c168-4f71-846b-eae0ed630bdf" containerID="6990ae8bdacb3a763e538dc2a08cd0b046bfa271f67476be05cca0af09d7adfa" exitCode=0 Sep 07 16:32:12.618642 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:12.618524 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1e4cf2b6-84cc97bbdb-h5sjl" event={"ID":"3aeff7e5-c168-4f71-846b-eae0ed630bdf","Type":"ContainerDied","Data":"6990ae8bdacb3a763e538dc2a08cd0b046bfa271f67476be05cca0af09d7adfa"} Sep 07 16:32:12.618642 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:12.618555 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1e4cf2b6-84cc97bbdb-h5sjl" Sep 07 16:32:12.618642 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:12.618578 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1e4cf2b6-84cc97bbdb-h5sjl" event={"ID":"3aeff7e5-c168-4f71-846b-eae0ed630bdf","Type":"ContainerDied","Data":"f54e82e05a94a054e15765eec06538c4feda2679f4964c2ce23d30da6af6d67b"} Sep 07 16:32:12.626859 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:12.626842 2580 scope.go:117] "RemoveContainer" containerID="7eda34e7bf31a9801b23c5e8433c8764213f5ad304fd5a3d9544a81dad2fe518" Sep 07 16:32:12.627227 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:32:12.627201 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7eda34e7bf31a9801b23c5e8433c8764213f5ad304fd5a3d9544a81dad2fe518\": container with ID starting with 7eda34e7bf31a9801b23c5e8433c8764213f5ad304fd5a3d9544a81dad2fe518 not found: ID does not exist" containerID="7eda34e7bf31a9801b23c5e8433c8764213f5ad304fd5a3d9544a81dad2fe518" Sep 07 16:32:12.627296 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:12.627240 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7eda34e7bf31a9801b23c5e8433c8764213f5ad304fd5a3d9544a81dad2fe518"} err="failed to get container status \"7eda34e7bf31a9801b23c5e8433c8764213f5ad304fd5a3d9544a81dad2fe518\": rpc error: code = NotFound desc = could not find container \"7eda34e7bf31a9801b23c5e8433c8764213f5ad304fd5a3d9544a81dad2fe518\": container with ID starting with 7eda34e7bf31a9801b23c5e8433c8764213f5ad304fd5a3d9544a81dad2fe518 not found: ID does not exist" Sep 07 16:32:12.627296 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:12.627263 2580 scope.go:117] "RemoveContainer" containerID="6990ae8bdacb3a763e538dc2a08cd0b046bfa271f67476be05cca0af09d7adfa" Sep 07 16:32:12.635317 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:12.635301 2580 scope.go:117] "RemoveContainer" containerID="6990ae8bdacb3a763e538dc2a08cd0b046bfa271f67476be05cca0af09d7adfa" Sep 07 16:32:12.635636 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:32:12.635616 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6990ae8bdacb3a763e538dc2a08cd0b046bfa271f67476be05cca0af09d7adfa\": container with ID starting with 6990ae8bdacb3a763e538dc2a08cd0b046bfa271f67476be05cca0af09d7adfa not found: ID does not exist" containerID="6990ae8bdacb3a763e538dc2a08cd0b046bfa271f67476be05cca0af09d7adfa" Sep 07 16:32:12.635684 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:12.635644 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6990ae8bdacb3a763e538dc2a08cd0b046bfa271f67476be05cca0af09d7adfa"} err="failed to get container status \"6990ae8bdacb3a763e538dc2a08cd0b046bfa271f67476be05cca0af09d7adfa\": rpc error: code = NotFound desc = could not find container \"6990ae8bdacb3a763e538dc2a08cd0b046bfa271f67476be05cca0af09d7adfa\": container with ID starting with 6990ae8bdacb3a763e538dc2a08cd0b046bfa271f67476be05cca0af09d7adfa not found: ID does not exist" Sep 07 16:32:12.639810 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:12.639788 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-1e4cf2b6-84cc97bbdb-h5sjl"] Sep 07 16:32:12.644956 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:12.644930 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-1e4cf2b6-84cc97bbdb-h5sjl"] Sep 07 16:32:12.653665 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:12.653643 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1e4cf2b6-f99494db6-cnw8z"] Sep 07 16:32:12.657198 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:12.657176 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1e4cf2b6-f99494db6-cnw8z"] Sep 07 16:32:12.967695 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:12.967659 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6"] Sep 07 16:32:12.968017 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:12.968003 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3aeff7e5-c168-4f71-846b-eae0ed630bdf" containerName="payload-pre-processing" Sep 07 16:32:12.968017 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:12.968045 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3aeff7e5-c168-4f71-846b-eae0ed630bdf" containerName="payload-pre-processing" Sep 07 16:32:12.968206 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:12.968058 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3624b12-0150-4711-8863-33bb7104662a" containerName="revoke-keys" Sep 07 16:32:12.968206 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:12.968065 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3624b12-0150-4711-8863-33bb7104662a" containerName="revoke-keys" Sep 07 16:32:12.968206 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:12.968087 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84d7d6ff-755a-4145-8c74-23c6e4af6332" containerName="payload-processing" Sep 07 16:32:12.968206 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:12.968092 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="84d7d6ff-755a-4145-8c74-23c6e4af6332" containerName="payload-processing" Sep 07 16:32:12.968206 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:12.968157 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6235888e-5e89-4892-b007-701682139ebd" containerName="istio-proxy" Sep 07 16:32:12.968206 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:12.968167 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="84d7d6ff-755a-4145-8c74-23c6e4af6332" containerName="payload-processing" Sep 07 16:32:12.968206 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:12.968174 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3624b12-0150-4711-8863-33bb7104662a" containerName="revoke-keys" Sep 07 16:32:12.968206 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:12.968182 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3aeff7e5-c168-4f71-846b-eae0ed630bdf" containerName="payload-pre-processing" Sep 07 16:32:12.972577 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:12.972555 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6" Sep 07 16:32:12.974807 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:12.974787 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-b1b87a00-openshift-default-dockercfg-nb6hp\"" Sep 07 16:32:12.982055 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:12.982007 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6"] Sep 07 16:32:13.011064 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:13.010998 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6-istio-token\") pod \"e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6\" (UID: \"7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6\") " pod="openshift-ingress/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6" Sep 07 16:32:13.011257 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:13.011080 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6-istio-podinfo\") pod \"e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6\" (UID: \"7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6\") " pod="openshift-ingress/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6" Sep 07 16:32:13.011257 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:13.011117 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6-istiod-ca-cert\") pod \"e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6\" (UID: \"7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6\") " pod="openshift-ingress/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6" Sep 07 16:32:13.011257 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:13.011149 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6-workload-socket\") pod \"e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6\" (UID: \"7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6\") " pod="openshift-ingress/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6" Sep 07 16:32:13.011257 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:13.011186 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6-credential-socket\") pod \"e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6\" (UID: \"7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6\") " pod="openshift-ingress/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6" Sep 07 16:32:13.011416 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:13.011312 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6-istio-envoy\") pod \"e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6\" (UID: \"7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6\") " pod="openshift-ingress/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6" Sep 07 16:32:13.011416 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:13.011341 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6-istio-data\") pod \"e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6\" (UID: \"7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6\") " pod="openshift-ingress/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6" Sep 07 16:32:13.011416 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:13.011360 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzjr5\" (UniqueName: \"kubernetes.io/projected/7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6-kube-api-access-lzjr5\") pod \"e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6\" (UID: \"7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6\") " pod="openshift-ingress/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6" Sep 07 16:32:13.011416 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:13.011388 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6-workload-certs\") pod \"e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6\" (UID: \"7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6\") " pod="openshift-ingress/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6" Sep 07 16:32:13.112410 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:13.112367 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6-istio-podinfo\") pod \"e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6\" (UID: \"7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6\") " pod="openshift-ingress/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6" Sep 07 16:32:13.112611 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:13.112425 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6-istiod-ca-cert\") pod \"e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6\" (UID: \"7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6\") " pod="openshift-ingress/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6" Sep 07 16:32:13.112611 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:13.112457 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6-workload-socket\") pod \"e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6\" (UID: \"7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6\") " pod="openshift-ingress/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6" Sep 07 16:32:13.112611 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:13.112494 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6-credential-socket\") pod \"e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6\" (UID: \"7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6\") " pod="openshift-ingress/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6" Sep 07 16:32:13.112611 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:13.112551 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6-istio-envoy\") pod \"e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6\" (UID: \"7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6\") " pod="openshift-ingress/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6" Sep 07 16:32:13.112611 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:13.112577 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6-istio-data\") pod \"e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6\" (UID: \"7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6\") " pod="openshift-ingress/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6" Sep 07 16:32:13.112611 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:13.112605 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzjr5\" (UniqueName: \"kubernetes.io/projected/7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6-kube-api-access-lzjr5\") pod \"e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6\" (UID: \"7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6\") " pod="openshift-ingress/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6" Sep 07 16:32:13.112892 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:13.112633 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6-workload-certs\") pod \"e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6\" (UID: \"7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6\") " pod="openshift-ingress/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6" Sep 07 16:32:13.112892 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:13.112703 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6-istio-token\") pod \"e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6\" (UID: \"7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6\") " pod="openshift-ingress/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6" Sep 07 16:32:13.113000 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:13.112935 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6-workload-socket\") pod \"e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6\" (UID: \"7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6\") " pod="openshift-ingress/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6" Sep 07 16:32:13.113145 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:13.113067 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6-istio-data\") pod \"e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6\" (UID: \"7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6\") " pod="openshift-ingress/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6" Sep 07 16:32:13.113284 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:13.113259 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6-workload-certs\") pod \"e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6\" (UID: \"7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6\") " pod="openshift-ingress/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6" Sep 07 16:32:13.113429 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:13.113357 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6-credential-socket\") pod \"e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6\" (UID: \"7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6\") " pod="openshift-ingress/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6" Sep 07 16:32:13.113532 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:13.113510 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6-istiod-ca-cert\") pod \"e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6\" (UID: \"7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6\") " pod="openshift-ingress/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6" Sep 07 16:32:13.115177 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:13.115156 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6-istio-envoy\") pod \"e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6\" (UID: \"7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6\") " pod="openshift-ingress/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6" Sep 07 16:32:13.115261 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:13.115224 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6-istio-podinfo\") pod \"e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6\" (UID: \"7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6\") " pod="openshift-ingress/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6" Sep 07 16:32:13.120135 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:13.120115 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzjr5\" (UniqueName: \"kubernetes.io/projected/7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6-kube-api-access-lzjr5\") pod \"e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6\" (UID: \"7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6\") " pod="openshift-ingress/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6" Sep 07 16:32:13.120229 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:13.120199 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6-istio-token\") pod \"e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6\" (UID: \"7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6\") " pod="openshift-ingress/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6" Sep 07 16:32:13.292439 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:13.292346 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6" Sep 07 16:32:13.419745 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:13.419710 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3aeff7e5-c168-4f71-846b-eae0ed630bdf" path="/var/lib/kubelet/pods/3aeff7e5-c168-4f71-846b-eae0ed630bdf/volumes" Sep 07 16:32:13.420207 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:13.420192 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84d7d6ff-755a-4145-8c74-23c6e4af6332" path="/var/lib/kubelet/pods/84d7d6ff-755a-4145-8c74-23c6e4af6332/volumes" Sep 07 16:32:13.420578 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:13.420565 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3624b12-0150-4711-8863-33bb7104662a" path="/var/lib/kubelet/pods/f3624b12-0150-4711-8863-33bb7104662a/volumes" Sep 07 16:32:13.569425 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:13.569394 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6"] Sep 07 16:32:13.572992 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:32:13.572962 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f896d0a_c7d9_49f0_b3be_a8a0b2e752a6.slice/crio-40d6335bfbb0bfeff57cec147088d00d8dbf15eb58399ab5d2f0d75383062836 WatchSource:0}: Error finding container 40d6335bfbb0bfeff57cec147088d00d8dbf15eb58399ab5d2f0d75383062836: Status 404 returned error can't find the container with id 40d6335bfbb0bfeff57cec147088d00d8dbf15eb58399ab5d2f0d75383062836 Sep 07 16:32:13.575683 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:13.575649 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:32:13.575798 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:13.575724 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:32:13.575798 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:13.575754 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:32:13.625315 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:13.625286 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6" event={"ID":"7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6","Type":"ContainerStarted","Data":"40d6335bfbb0bfeff57cec147088d00d8dbf15eb58399ab5d2f0d75383062836"} Sep 07 16:32:14.633443 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:14.633399 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6" event={"ID":"7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6","Type":"ContainerStarted","Data":"edec555a37ef75ecb641b3af59c808f19bf5bd15cfa1ea0a3e8df2c15bc1004e"} Sep 07 16:32:14.651939 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:14.651886 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6" podStartSLOduration=2.651869499 podStartE2EDuration="2.651869499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:32:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:32:14.650304816 +0000 UTC m=+763.685527885" watchObservedRunningTime="2026-09-07 16:32:14.651869499 +0000 UTC m=+763.687092567" Sep 07 16:32:15.292559 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:15.292515 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6" Sep 07 16:32:15.297754 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:15.297731 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6" Sep 07 16:32:15.638142 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:15.638050 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6" Sep 07 16:32:15.639235 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:15.639217 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6" Sep 07 16:32:17.822050 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:17.821990 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b1b87a00-fc89df65d-264lb"] Sep 07 16:32:17.825432 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:17.825416 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b1b87a00-fc89df65d-264lb" Sep 07 16:32:17.827464 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:17.827441 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-b1b87a00\"" Sep 07 16:32:17.827568 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:17.827477 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-b1b87a00-dockercfg-9t4wb\"" Sep 07 16:32:17.836182 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:17.836153 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b1b87a00-fc89df65d-264lb"] Sep 07 16:32:17.853809 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:17.853775 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blgg2\" (UniqueName: \"kubernetes.io/projected/22f4186f-7df5-4d29-a517-e4201be8ca29-kube-api-access-blgg2\") pod \"payload-pre-processing-e2e-ait-b1b87a00-fc89df65d-264lb\" (UID: \"22f4186f-7df5-4d29-a517-e4201be8ca29\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b1b87a00-fc89df65d-264lb" Sep 07 16:32:17.854078 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:17.854014 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22f4186f-7df5-4d29-a517-e4201be8ca29-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b1b87a00-fc89df65d-264lb\" (UID: \"22f4186f-7df5-4d29-a517-e4201be8ca29\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b1b87a00-fc89df65d-264lb" Sep 07 16:32:17.955338 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:17.955300 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blgg2\" (UniqueName: \"kubernetes.io/projected/22f4186f-7df5-4d29-a517-e4201be8ca29-kube-api-access-blgg2\") pod \"payload-pre-processing-e2e-ait-b1b87a00-fc89df65d-264lb\" (UID: \"22f4186f-7df5-4d29-a517-e4201be8ca29\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b1b87a00-fc89df65d-264lb" Sep 07 16:32:17.955338 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:17.955338 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22f4186f-7df5-4d29-a517-e4201be8ca29-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b1b87a00-fc89df65d-264lb\" (UID: \"22f4186f-7df5-4d29-a517-e4201be8ca29\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b1b87a00-fc89df65d-264lb" Sep 07 16:32:17.956005 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:17.955987 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22f4186f-7df5-4d29-a517-e4201be8ca29-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b1b87a00-fc89df65d-264lb\" (UID: \"22f4186f-7df5-4d29-a517-e4201be8ca29\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b1b87a00-fc89df65d-264lb" Sep 07 16:32:17.962656 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:17.962629 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blgg2\" (UniqueName: \"kubernetes.io/projected/22f4186f-7df5-4d29-a517-e4201be8ca29-kube-api-access-blgg2\") pod \"payload-pre-processing-e2e-ait-b1b87a00-fc89df65d-264lb\" (UID: \"22f4186f-7df5-4d29-a517-e4201be8ca29\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b1b87a00-fc89df65d-264lb" Sep 07 16:32:18.137918 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:18.137818 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b1b87a00-fc89df65d-264lb" Sep 07 16:32:18.411993 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:18.411932 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b1b87a00-fc89df65d-264lb"] Sep 07 16:32:18.414524 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:32:18.414496 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22f4186f_7df5_4d29_a517_e4201be8ca29.slice/crio-fbb34fda1fab252e7d9fcbe9e31dd43b41cbf1d71a8237cb15da30b584572882 WatchSource:0}: Error finding container fbb34fda1fab252e7d9fcbe9e31dd43b41cbf1d71a8237cb15da30b584572882: Status 404 returned error can't find the container with id fbb34fda1fab252e7d9fcbe9e31dd43b41cbf1d71a8237cb15da30b584572882 Sep 07 16:32:18.652992 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:18.652956 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b1b87a00-fc89df65d-264lb" event={"ID":"22f4186f-7df5-4d29-a517-e4201be8ca29","Type":"ContainerStarted","Data":"fff4b0be4bc3ea7056157853603f26f4e74c79630e05a2d476921b87b2557758"} Sep 07 16:32:18.653190 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:18.653036 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b1b87a00-fc89df65d-264lb" event={"ID":"22f4186f-7df5-4d29-a517-e4201be8ca29","Type":"ContainerStarted","Data":"fbb34fda1fab252e7d9fcbe9e31dd43b41cbf1d71a8237cb15da30b584572882"} Sep 07 16:32:18.653190 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:18.653067 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b1b87a00-fc89df65d-264lb" Sep 07 16:32:18.667854 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:18.667756 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-b1b87a00-fc89df65d-264lb" podStartSLOduration=1.667739163 podStartE2EDuration="1.667739163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:32:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:32:18.666187321 +0000 UTC m=+767.701410393" watchObservedRunningTime="2026-09-07 16:32:18.667739163 +0000 UTC m=+767.702962233" Sep 07 16:32:29.660698 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:29.660663 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b1b87a00-fc89df65d-264lb" Sep 07 16:32:31.680238 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:31.680147 2580 patch_prober.go:28] interesting pod/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 07 16:32:31.680674 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:31.680234 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6" podUID="7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 07 16:32:32.722520 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:32.722483 2580 generic.go:356] "Generic (PLEG): container finished" podID="7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6" containerID="edec555a37ef75ecb641b3af59c808f19bf5bd15cfa1ea0a3e8df2c15bc1004e" exitCode=0 Sep 07 16:32:32.722933 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:32.722535 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6" event={"ID":"7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6","Type":"ContainerDied","Data":"edec555a37ef75ecb641b3af59c808f19bf5bd15cfa1ea0a3e8df2c15bc1004e"} Sep 07 16:32:32.722988 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:32.722947 2580 scope.go:117] "RemoveContainer" containerID="edec555a37ef75ecb641b3af59c808f19bf5bd15cfa1ea0a3e8df2c15bc1004e" Sep 07 16:32:32.724072 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:32.724047 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:32:32.724127 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:32.724111 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:32:32.724160 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:32.724141 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:32:33.732185 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:33.732145 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6" event={"ID":"7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6","Type":"ContainerStarted","Data":"9739a13c160f480593f5c8549a3036eab1dab58f26f30d32c1abdca155d1c61a"} Sep 07 16:32:34.293339 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:34.293305 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6" Sep 07 16:32:34.293582 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:34.293559 2580 patch_prober.go:28] interesting pod/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Sep 07 16:32:34.293657 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:34.293616 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6" podUID="7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Sep 07 16:32:35.293627 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:35.293586 2580 patch_prober.go:28] interesting pod/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Sep 07 16:32:35.294087 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:35.293659 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6" podUID="7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Sep 07 16:32:36.293570 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:36.293526 2580 patch_prober.go:28] interesting pod/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Sep 07 16:32:36.293754 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:36.293605 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6" podUID="7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Sep 07 16:32:37.293530 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:37.293492 2580 patch_prober.go:28] interesting pod/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Sep 07 16:32:37.293713 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:37.293554 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6" podUID="7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Sep 07 16:32:38.293418 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:38.293381 2580 patch_prober.go:28] interesting pod/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Sep 07 16:32:38.293788 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:38.293436 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6" podUID="7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Sep 07 16:32:39.296770 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:39.296735 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6" Sep 07 16:32:39.297167 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:39.297008 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6" Sep 07 16:32:39.297635 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:39.297616 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6" Sep 07 16:32:41.595581 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:41.595551 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b1b87a00-hxt7n"] Sep 07 16:32:41.598197 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:41.598179 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b1b87a00-hxt7n" Sep 07 16:32:41.600289 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:41.600269 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 07 16:32:41.600404 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:41.600324 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jpjxt\"" Sep 07 16:32:41.605847 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:41.605313 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b1b87a00-hxt7n"] Sep 07 16:32:41.691652 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:41.691605 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0599c52e-98ca-4420-bb61-4eb63a18a5e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b1b87a00-hxt7n\" (UID: \"0599c52e-98ca-4420-bb61-4eb63a18a5e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b1b87a00-hxt7n" Sep 07 16:32:41.792855 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:41.792815 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0599c52e-98ca-4420-bb61-4eb63a18a5e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b1b87a00-hxt7n\" (UID: \"0599c52e-98ca-4420-bb61-4eb63a18a5e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b1b87a00-hxt7n" Sep 07 16:32:41.793467 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:41.793447 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0599c52e-98ca-4420-bb61-4eb63a18a5e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b1b87a00-hxt7n\" (UID: \"0599c52e-98ca-4420-bb61-4eb63a18a5e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b1b87a00-hxt7n" Sep 07 16:32:41.910256 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:41.910172 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b1b87a00-hxt7n" Sep 07 16:32:42.175508 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:42.175469 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b1b87a00-hxt7n"] Sep 07 16:32:42.177914 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:32:42.177886 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0599c52e_98ca_4420_bb61_4eb63a18a5e7.slice/crio-21dac3520c6a90d8db4de7059363c49c67742c50728416ed4a9eaaa3445ce8b8 WatchSource:0}: Error finding container 21dac3520c6a90d8db4de7059363c49c67742c50728416ed4a9eaaa3445ce8b8: Status 404 returned error can't find the container with id 21dac3520c6a90d8db4de7059363c49c67742c50728416ed4a9eaaa3445ce8b8 Sep 07 16:32:42.776006 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:42.775920 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b1b87a00-hxt7n" event={"ID":"0599c52e-98ca-4420-bb61-4eb63a18a5e7","Type":"ContainerStarted","Data":"0b2bc20721af3fabbdf3234324202c60ef21be12c10fddbf565fb4404516a5ed"} Sep 07 16:32:42.776006 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:42.775972 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b1b87a00-hxt7n" event={"ID":"0599c52e-98ca-4420-bb61-4eb63a18a5e7","Type":"ContainerStarted","Data":"21dac3520c6a90d8db4de7059363c49c67742c50728416ed4a9eaaa3445ce8b8"} Sep 07 16:32:42.787480 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:42.787432 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b1b87a00-hxt7n" podStartSLOduration=1.787416093 podStartE2EDuration="1.787416093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:32:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:32:42.78658753 +0000 UTC m=+791.821810614" watchObservedRunningTime="2026-09-07 16:32:42.787416093 +0000 UTC m=+791.822639163" Sep 07 16:32:43.781828 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:43.781790 2580 generic.go:356] "Generic (PLEG): container finished" podID="0599c52e-98ca-4420-bb61-4eb63a18a5e7" containerID="0b2bc20721af3fabbdf3234324202c60ef21be12c10fddbf565fb4404516a5ed" exitCode=0 Sep 07 16:32:43.782255 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:43.781858 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b1b87a00-hxt7n" event={"ID":"0599c52e-98ca-4420-bb61-4eb63a18a5e7","Type":"ContainerDied","Data":"0b2bc20721af3fabbdf3234324202c60ef21be12c10fddbf565fb4404516a5ed"} Sep 07 16:32:44.925804 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:44.925779 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b1b87a00-hxt7n" Sep 07 16:32:45.122506 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:45.122403 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0599c52e-98ca-4420-bb61-4eb63a18a5e7-maas-api-service-ca\") pod \"0599c52e-98ca-4420-bb61-4eb63a18a5e7\" (UID: \"0599c52e-98ca-4420-bb61-4eb63a18a5e7\") " Sep 07 16:32:45.123096 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:45.123054 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0599c52e-98ca-4420-bb61-4eb63a18a5e7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0599c52e-98ca-4420-bb61-4eb63a18a5e7" (UID: "0599c52e-98ca-4420-bb61-4eb63a18a5e7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 16:32:45.223871 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:45.223832 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0599c52e-98ca-4420-bb61-4eb63a18a5e7-maas-api-service-ca\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:32:45.792711 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:45.792678 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b1b87a00-hxt7n" event={"ID":"0599c52e-98ca-4420-bb61-4eb63a18a5e7","Type":"ContainerDied","Data":"21dac3520c6a90d8db4de7059363c49c67742c50728416ed4a9eaaa3445ce8b8"} Sep 07 16:32:45.792873 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:45.792729 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21dac3520c6a90d8db4de7059363c49c67742c50728416ed4a9eaaa3445ce8b8" Sep 07 16:32:45.792873 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:45.792689 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b1b87a00-hxt7n" Sep 07 16:32:46.374608 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:46.374572 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8a3403-k6xdr"] Sep 07 16:32:46.375676 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:46.375649 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0599c52e-98ca-4420-bb61-4eb63a18a5e7" containerName="revoke-keys" Sep 07 16:32:46.375837 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:46.375821 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0599c52e-98ca-4420-bb61-4eb63a18a5e7" containerName="revoke-keys" Sep 07 16:32:46.376105 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:46.376087 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0599c52e-98ca-4420-bb61-4eb63a18a5e7" containerName="revoke-keys" Sep 07 16:32:46.378706 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:46.378688 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8a3403-k6xdr" Sep 07 16:32:46.381728 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:46.381531 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 07 16:32:46.381728 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:46.381599 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jpjxt\"" Sep 07 16:32:46.384405 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:46.384381 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8a3403-k6xdr"] Sep 07 16:32:46.433315 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:46.433285 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4837f09b-32a8-4661-937d-323369bcbb03-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-8a3403-k6xdr\" (UID: \"4837f09b-32a8-4661-937d-323369bcbb03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8a3403-k6xdr" Sep 07 16:32:46.533709 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:46.533671 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4837f09b-32a8-4661-937d-323369bcbb03-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-8a3403-k6xdr\" (UID: \"4837f09b-32a8-4661-937d-323369bcbb03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8a3403-k6xdr" Sep 07 16:32:46.534397 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:46.534359 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4837f09b-32a8-4661-937d-323369bcbb03-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-8a3403-k6xdr\" (UID: \"4837f09b-32a8-4661-937d-323369bcbb03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8a3403-k6xdr" Sep 07 16:32:46.695377 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:46.695339 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8a3403-k6xdr" Sep 07 16:32:46.965134 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:46.965061 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8a3403-k6xdr"] Sep 07 16:32:46.968460 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:32:46.968430 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4837f09b_32a8_4661_937d_323369bcbb03.slice/crio-0874c7fbf6445a7af3ca20965f24d9ea03289ca50cd49c53dc1fd7f1ed6852e2 WatchSource:0}: Error finding container 0874c7fbf6445a7af3ca20965f24d9ea03289ca50cd49c53dc1fd7f1ed6852e2: Status 404 returned error can't find the container with id 0874c7fbf6445a7af3ca20965f24d9ea03289ca50cd49c53dc1fd7f1ed6852e2 Sep 07 16:32:47.592620 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:47.592587 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b1b87a00-fc89df65d-264lb"] Sep 07 16:32:47.593097 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:47.592847 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-b1b87a00-fc89df65d-264lb" podUID="22f4186f-7df5-4d29-a517-e4201be8ca29" containerName="payload-pre-processing" containerID="cri-o://fff4b0be4bc3ea7056157853603f26f4e74c79630e05a2d476921b87b2557758" gracePeriod=30 Sep 07 16:32:47.749318 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:47.749282 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b1b87a00-hxt7n"] Sep 07 16:32:47.752833 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:47.752803 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b1b87a00-hxt7n"] Sep 07 16:32:47.804240 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:47.804208 2580 generic.go:356] "Generic (PLEG): container finished" podID="4837f09b-32a8-4661-937d-323369bcbb03" containerID="d582e5fd6fba39ee1dae0f6455cd6097f788d90739a32791531a31d44c42efb4" exitCode=0 Sep 07 16:32:47.804425 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:47.804240 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8a3403-k6xdr" event={"ID":"4837f09b-32a8-4661-937d-323369bcbb03","Type":"ContainerDied","Data":"d582e5fd6fba39ee1dae0f6455cd6097f788d90739a32791531a31d44c42efb4"} Sep 07 16:32:47.804425 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:47.804311 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8a3403-k6xdr" event={"ID":"4837f09b-32a8-4661-937d-323369bcbb03","Type":"ContainerStarted","Data":"0874c7fbf6445a7af3ca20965f24d9ea03289ca50cd49c53dc1fd7f1ed6852e2"} Sep 07 16:32:47.805911 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:47.805889 2580 generic.go:356] "Generic (PLEG): container finished" podID="22f4186f-7df5-4d29-a517-e4201be8ca29" containerID="fff4b0be4bc3ea7056157853603f26f4e74c79630e05a2d476921b87b2557758" exitCode=0 Sep 07 16:32:47.806037 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:47.805927 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b1b87a00-fc89df65d-264lb" event={"ID":"22f4186f-7df5-4d29-a517-e4201be8ca29","Type":"ContainerDied","Data":"fff4b0be4bc3ea7056157853603f26f4e74c79630e05a2d476921b87b2557758"} Sep 07 16:32:47.875683 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:47.875658 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b1b87a00-fc89df65d-264lb" Sep 07 16:32:47.944619 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:47.944586 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-blgg2\" (UniqueName: \"kubernetes.io/projected/22f4186f-7df5-4d29-a517-e4201be8ca29-kube-api-access-blgg2\") pod \"22f4186f-7df5-4d29-a517-e4201be8ca29\" (UID: \"22f4186f-7df5-4d29-a517-e4201be8ca29\") " Sep 07 16:32:47.944805 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:47.944637 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22f4186f-7df5-4d29-a517-e4201be8ca29-plugins-config-volume\") pod \"22f4186f-7df5-4d29-a517-e4201be8ca29\" (UID: \"22f4186f-7df5-4d29-a517-e4201be8ca29\") " Sep 07 16:32:47.945052 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:47.944994 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22f4186f-7df5-4d29-a517-e4201be8ca29-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "22f4186f-7df5-4d29-a517-e4201be8ca29" (UID: "22f4186f-7df5-4d29-a517-e4201be8ca29"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 16:32:47.946974 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:47.946956 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22f4186f-7df5-4d29-a517-e4201be8ca29-kube-api-access-blgg2" (OuterVolumeSpecName: "kube-api-access-blgg2") pod "22f4186f-7df5-4d29-a517-e4201be8ca29" (UID: "22f4186f-7df5-4d29-a517-e4201be8ca29"). InnerVolumeSpecName "kube-api-access-blgg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 16:32:48.045408 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:48.045375 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22f4186f-7df5-4d29-a517-e4201be8ca29-plugins-config-volume\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:32:48.045408 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:48.045403 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blgg2\" (UniqueName: \"kubernetes.io/projected/22f4186f-7df5-4d29-a517-e4201be8ca29-kube-api-access-blgg2\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:32:48.812431 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:48.812403 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b1b87a00-fc89df65d-264lb" Sep 07 16:32:48.812431 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:48.812414 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b1b87a00-fc89df65d-264lb" event={"ID":"22f4186f-7df5-4d29-a517-e4201be8ca29","Type":"ContainerDied","Data":"fbb34fda1fab252e7d9fcbe9e31dd43b41cbf1d71a8237cb15da30b584572882"} Sep 07 16:32:48.812911 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:48.812488 2580 scope.go:117] "RemoveContainer" containerID="fff4b0be4bc3ea7056157853603f26f4e74c79630e05a2d476921b87b2557758" Sep 07 16:32:48.817832 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:48.817781 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6"] Sep 07 16:32:48.818196 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:48.818143 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6" podUID="7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6" containerName="istio-proxy" containerID="cri-o://9739a13c160f480593f5c8549a3036eab1dab58f26f30d32c1abdca155d1c61a" gracePeriod=30 Sep 07 16:32:48.851114 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:48.851080 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b1b87a00-fc89df65d-264lb"] Sep 07 16:32:48.853350 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:48.853321 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b1b87a00-fc89df65d-264lb"] Sep 07 16:32:48.950752 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:48.950729 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8a3403-k6xdr" Sep 07 16:32:49.054229 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:49.054193 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4837f09b-32a8-4661-937d-323369bcbb03-maas-api-service-ca\") pod \"4837f09b-32a8-4661-937d-323369bcbb03\" (UID: \"4837f09b-32a8-4661-937d-323369bcbb03\") " Sep 07 16:32:49.054586 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:49.054564 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4837f09b-32a8-4661-937d-323369bcbb03-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4837f09b-32a8-4661-937d-323369bcbb03" (UID: "4837f09b-32a8-4661-937d-323369bcbb03"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 16:32:49.155702 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:49.155616 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4837f09b-32a8-4661-937d-323369bcbb03-maas-api-service-ca\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:32:49.418137 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:49.418065 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0599c52e-98ca-4420-bb61-4eb63a18a5e7" path="/var/lib/kubelet/pods/0599c52e-98ca-4420-bb61-4eb63a18a5e7/volumes" Sep 07 16:32:49.418478 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:49.418464 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22f4186f-7df5-4d29-a517-e4201be8ca29" path="/var/lib/kubelet/pods/22f4186f-7df5-4d29-a517-e4201be8ca29/volumes" Sep 07 16:32:49.486852 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:49.486818 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-39437224-openshift-default-6db78586d-dpsvs"] Sep 07 16:32:49.487185 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:49.487171 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4837f09b-32a8-4661-937d-323369bcbb03" containerName="revoke-keys" Sep 07 16:32:49.487263 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:49.487186 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4837f09b-32a8-4661-937d-323369bcbb03" containerName="revoke-keys" Sep 07 16:32:49.487263 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:49.487199 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22f4186f-7df5-4d29-a517-e4201be8ca29" containerName="payload-pre-processing" Sep 07 16:32:49.487263 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:49.487204 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="22f4186f-7df5-4d29-a517-e4201be8ca29" containerName="payload-pre-processing" Sep 07 16:32:49.487263 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:49.487262 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4837f09b-32a8-4661-937d-323369bcbb03" containerName="revoke-keys" Sep 07 16:32:49.487463 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:49.487269 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="22f4186f-7df5-4d29-a517-e4201be8ca29" containerName="payload-pre-processing" Sep 07 16:32:49.489606 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:49.489584 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-39437224-openshift-default-6db78586d-dpsvs" Sep 07 16:32:49.491743 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:49.491717 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-39437224-openshift-default-dockercfg-vhgcn\"" Sep 07 16:32:49.502777 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:49.502744 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-39437224-openshift-default-6db78586d-dpsvs"] Sep 07 16:32:49.559042 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:49.558998 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e124b934-cb8e-485f-8bec-4348c1307769-istio-envoy\") pod \"e2e-derive-39437224-openshift-default-6db78586d-dpsvs\" (UID: \"e124b934-cb8e-485f-8bec-4348c1307769\") " pod="openshift-ingress/e2e-derive-39437224-openshift-default-6db78586d-dpsvs" Sep 07 16:32:49.559298 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:49.559077 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e124b934-cb8e-485f-8bec-4348c1307769-workload-socket\") pod \"e2e-derive-39437224-openshift-default-6db78586d-dpsvs\" (UID: \"e124b934-cb8e-485f-8bec-4348c1307769\") " pod="openshift-ingress/e2e-derive-39437224-openshift-default-6db78586d-dpsvs" Sep 07 16:32:49.559298 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:49.559128 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e124b934-cb8e-485f-8bec-4348c1307769-istio-token\") pod \"e2e-derive-39437224-openshift-default-6db78586d-dpsvs\" (UID: \"e124b934-cb8e-485f-8bec-4348c1307769\") " pod="openshift-ingress/e2e-derive-39437224-openshift-default-6db78586d-dpsvs" Sep 07 16:32:49.559298 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:49.559216 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e124b934-cb8e-485f-8bec-4348c1307769-istiod-ca-cert\") pod \"e2e-derive-39437224-openshift-default-6db78586d-dpsvs\" (UID: \"e124b934-cb8e-485f-8bec-4348c1307769\") " pod="openshift-ingress/e2e-derive-39437224-openshift-default-6db78586d-dpsvs" Sep 07 16:32:49.559298 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:49.559259 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e124b934-cb8e-485f-8bec-4348c1307769-credential-socket\") pod \"e2e-derive-39437224-openshift-default-6db78586d-dpsvs\" (UID: \"e124b934-cb8e-485f-8bec-4348c1307769\") " pod="openshift-ingress/e2e-derive-39437224-openshift-default-6db78586d-dpsvs" Sep 07 16:32:49.559298 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:49.559285 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e124b934-cb8e-485f-8bec-4348c1307769-istio-podinfo\") pod \"e2e-derive-39437224-openshift-default-6db78586d-dpsvs\" (UID: \"e124b934-cb8e-485f-8bec-4348c1307769\") " pod="openshift-ingress/e2e-derive-39437224-openshift-default-6db78586d-dpsvs" Sep 07 16:32:49.559601 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:49.559321 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjx8v\" (UniqueName: \"kubernetes.io/projected/e124b934-cb8e-485f-8bec-4348c1307769-kube-api-access-jjx8v\") pod \"e2e-derive-39437224-openshift-default-6db78586d-dpsvs\" (UID: \"e124b934-cb8e-485f-8bec-4348c1307769\") " pod="openshift-ingress/e2e-derive-39437224-openshift-default-6db78586d-dpsvs" Sep 07 16:32:49.559601 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:49.559390 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e124b934-cb8e-485f-8bec-4348c1307769-workload-certs\") pod \"e2e-derive-39437224-openshift-default-6db78586d-dpsvs\" (UID: \"e124b934-cb8e-485f-8bec-4348c1307769\") " pod="openshift-ingress/e2e-derive-39437224-openshift-default-6db78586d-dpsvs" Sep 07 16:32:49.559601 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:49.559448 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e124b934-cb8e-485f-8bec-4348c1307769-istio-data\") pod \"e2e-derive-39437224-openshift-default-6db78586d-dpsvs\" (UID: \"e124b934-cb8e-485f-8bec-4348c1307769\") " pod="openshift-ingress/e2e-derive-39437224-openshift-default-6db78586d-dpsvs" Sep 07 16:32:49.660495 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:49.660457 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e124b934-cb8e-485f-8bec-4348c1307769-istio-token\") pod \"e2e-derive-39437224-openshift-default-6db78586d-dpsvs\" (UID: \"e124b934-cb8e-485f-8bec-4348c1307769\") " pod="openshift-ingress/e2e-derive-39437224-openshift-default-6db78586d-dpsvs" Sep 07 16:32:49.660692 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:49.660528 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e124b934-cb8e-485f-8bec-4348c1307769-istiod-ca-cert\") pod \"e2e-derive-39437224-openshift-default-6db78586d-dpsvs\" (UID: \"e124b934-cb8e-485f-8bec-4348c1307769\") " pod="openshift-ingress/e2e-derive-39437224-openshift-default-6db78586d-dpsvs" Sep 07 16:32:49.660692 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:49.660572 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e124b934-cb8e-485f-8bec-4348c1307769-credential-socket\") pod \"e2e-derive-39437224-openshift-default-6db78586d-dpsvs\" (UID: \"e124b934-cb8e-485f-8bec-4348c1307769\") " pod="openshift-ingress/e2e-derive-39437224-openshift-default-6db78586d-dpsvs" Sep 07 16:32:49.660692 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:49.660594 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e124b934-cb8e-485f-8bec-4348c1307769-istio-podinfo\") pod \"e2e-derive-39437224-openshift-default-6db78586d-dpsvs\" (UID: \"e124b934-cb8e-485f-8bec-4348c1307769\") " pod="openshift-ingress/e2e-derive-39437224-openshift-default-6db78586d-dpsvs" Sep 07 16:32:49.660692 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:49.660615 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjx8v\" (UniqueName: \"kubernetes.io/projected/e124b934-cb8e-485f-8bec-4348c1307769-kube-api-access-jjx8v\") pod \"e2e-derive-39437224-openshift-default-6db78586d-dpsvs\" (UID: \"e124b934-cb8e-485f-8bec-4348c1307769\") " pod="openshift-ingress/e2e-derive-39437224-openshift-default-6db78586d-dpsvs" Sep 07 16:32:49.660692 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:49.660662 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e124b934-cb8e-485f-8bec-4348c1307769-workload-certs\") pod \"e2e-derive-39437224-openshift-default-6db78586d-dpsvs\" (UID: \"e124b934-cb8e-485f-8bec-4348c1307769\") " pod="openshift-ingress/e2e-derive-39437224-openshift-default-6db78586d-dpsvs" Sep 07 16:32:49.660957 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:49.660712 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e124b934-cb8e-485f-8bec-4348c1307769-istio-data\") pod \"e2e-derive-39437224-openshift-default-6db78586d-dpsvs\" (UID: \"e124b934-cb8e-485f-8bec-4348c1307769\") " pod="openshift-ingress/e2e-derive-39437224-openshift-default-6db78586d-dpsvs" Sep 07 16:32:49.660957 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:49.660823 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e124b934-cb8e-485f-8bec-4348c1307769-istio-envoy\") pod \"e2e-derive-39437224-openshift-default-6db78586d-dpsvs\" (UID: \"e124b934-cb8e-485f-8bec-4348c1307769\") " pod="openshift-ingress/e2e-derive-39437224-openshift-default-6db78586d-dpsvs" Sep 07 16:32:49.660957 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:49.660854 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e124b934-cb8e-485f-8bec-4348c1307769-workload-socket\") pod \"e2e-derive-39437224-openshift-default-6db78586d-dpsvs\" (UID: \"e124b934-cb8e-485f-8bec-4348c1307769\") " pod="openshift-ingress/e2e-derive-39437224-openshift-default-6db78586d-dpsvs" Sep 07 16:32:49.661334 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:49.661306 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e124b934-cb8e-485f-8bec-4348c1307769-workload-socket\") pod \"e2e-derive-39437224-openshift-default-6db78586d-dpsvs\" (UID: \"e124b934-cb8e-485f-8bec-4348c1307769\") " pod="openshift-ingress/e2e-derive-39437224-openshift-default-6db78586d-dpsvs" Sep 07 16:32:49.661913 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:49.661887 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e124b934-cb8e-485f-8bec-4348c1307769-workload-certs\") pod \"e2e-derive-39437224-openshift-default-6db78586d-dpsvs\" (UID: \"e124b934-cb8e-485f-8bec-4348c1307769\") " pod="openshift-ingress/e2e-derive-39437224-openshift-default-6db78586d-dpsvs" Sep 07 16:32:49.662390 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:49.662075 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e124b934-cb8e-485f-8bec-4348c1307769-credential-socket\") pod \"e2e-derive-39437224-openshift-default-6db78586d-dpsvs\" (UID: \"e124b934-cb8e-485f-8bec-4348c1307769\") " pod="openshift-ingress/e2e-derive-39437224-openshift-default-6db78586d-dpsvs" Sep 07 16:32:49.662390 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:49.662170 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e124b934-cb8e-485f-8bec-4348c1307769-istio-data\") pod \"e2e-derive-39437224-openshift-default-6db78586d-dpsvs\" (UID: \"e124b934-cb8e-485f-8bec-4348c1307769\") " pod="openshift-ingress/e2e-derive-39437224-openshift-default-6db78586d-dpsvs" Sep 07 16:32:49.662390 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:49.662350 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e124b934-cb8e-485f-8bec-4348c1307769-istiod-ca-cert\") pod \"e2e-derive-39437224-openshift-default-6db78586d-dpsvs\" (UID: \"e124b934-cb8e-485f-8bec-4348c1307769\") " pod="openshift-ingress/e2e-derive-39437224-openshift-default-6db78586d-dpsvs" Sep 07 16:32:49.665277 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:49.665251 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e124b934-cb8e-485f-8bec-4348c1307769-istio-envoy\") pod \"e2e-derive-39437224-openshift-default-6db78586d-dpsvs\" (UID: \"e124b934-cb8e-485f-8bec-4348c1307769\") " pod="openshift-ingress/e2e-derive-39437224-openshift-default-6db78586d-dpsvs" Sep 07 16:32:49.665582 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:49.665561 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e124b934-cb8e-485f-8bec-4348c1307769-istio-podinfo\") pod \"e2e-derive-39437224-openshift-default-6db78586d-dpsvs\" (UID: \"e124b934-cb8e-485f-8bec-4348c1307769\") " pod="openshift-ingress/e2e-derive-39437224-openshift-default-6db78586d-dpsvs" Sep 07 16:32:49.672652 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:49.672595 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e124b934-cb8e-485f-8bec-4348c1307769-istio-token\") pod \"e2e-derive-39437224-openshift-default-6db78586d-dpsvs\" (UID: \"e124b934-cb8e-485f-8bec-4348c1307769\") " pod="openshift-ingress/e2e-derive-39437224-openshift-default-6db78586d-dpsvs" Sep 07 16:32:49.672830 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:49.672806 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjx8v\" (UniqueName: \"kubernetes.io/projected/e124b934-cb8e-485f-8bec-4348c1307769-kube-api-access-jjx8v\") pod \"e2e-derive-39437224-openshift-default-6db78586d-dpsvs\" (UID: \"e124b934-cb8e-485f-8bec-4348c1307769\") " pod="openshift-ingress/e2e-derive-39437224-openshift-default-6db78586d-dpsvs" Sep 07 16:32:49.805601 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:49.805554 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-39437224-openshift-default-6db78586d-dpsvs" Sep 07 16:32:49.818694 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:49.818654 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8a3403-k6xdr" event={"ID":"4837f09b-32a8-4661-937d-323369bcbb03","Type":"ContainerDied","Data":"0874c7fbf6445a7af3ca20965f24d9ea03289ca50cd49c53dc1fd7f1ed6852e2"} Sep 07 16:32:49.819098 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:49.818716 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0874c7fbf6445a7af3ca20965f24d9ea03289ca50cd49c53dc1fd7f1ed6852e2" Sep 07 16:32:49.819098 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:49.818666 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8a3403-k6xdr" Sep 07 16:32:50.080318 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:50.080287 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-39437224-openshift-default-6db78586d-dpsvs"] Sep 07 16:32:50.083408 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:32:50.083380 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode124b934_cb8e_485f_8bec_4348c1307769.slice/crio-174ea7dfdf5e8427ff34f0ab3c3e4525d216d76c7ce0e01199dff153e43548af WatchSource:0}: Error finding container 174ea7dfdf5e8427ff34f0ab3c3e4525d216d76c7ce0e01199dff153e43548af: Status 404 returned error can't find the container with id 174ea7dfdf5e8427ff34f0ab3c3e4525d216d76c7ce0e01199dff153e43548af Sep 07 16:32:50.085871 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:50.085840 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:32:50.085956 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:50.085900 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:32:50.085956 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:50.085931 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:32:50.826723 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:50.826687 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-39437224-openshift-default-6db78586d-dpsvs" event={"ID":"e124b934-cb8e-485f-8bec-4348c1307769","Type":"ContainerStarted","Data":"e183122982d767d924eac0637c007323eed070b282c4ef731db6bb4655d1e40e"} Sep 07 16:32:50.827146 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:50.826744 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-39437224-openshift-default-6db78586d-dpsvs" event={"ID":"e124b934-cb8e-485f-8bec-4348c1307769","Type":"ContainerStarted","Data":"174ea7dfdf5e8427ff34f0ab3c3e4525d216d76c7ce0e01199dff153e43548af"} Sep 07 16:32:50.849049 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:50.848961 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-39437224-openshift-default-6db78586d-dpsvs" podStartSLOduration=1.848939678 podStartE2EDuration="1.848939678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:32:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:32:50.846887699 +0000 UTC m=+799.882110770" watchObservedRunningTime="2026-09-07 16:32:50.848939678 +0000 UTC m=+799.884162748" Sep 07 16:32:51.806124 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:51.806084 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-39437224-openshift-default-6db78586d-dpsvs" Sep 07 16:32:51.811424 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:51.811399 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-39437224-openshift-default-6db78586d-dpsvs" Sep 07 16:32:51.834126 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:51.834094 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-39437224-openshift-default-6db78586d-dpsvs" Sep 07 16:32:51.837401 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:51.837378 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-39437224-openshift-default-6db78586d-dpsvs" Sep 07 16:32:54.071347 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.071321 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6" Sep 07 16:32:54.105135 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.105053 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6-workload-socket\") pod \"7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6\" (UID: \"7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6\") " Sep 07 16:32:54.105135 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.105103 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6-istio-token\") pod \"7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6\" (UID: \"7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6\") " Sep 07 16:32:54.105135 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.105124 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6-istio-data\") pod \"7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6\" (UID: \"7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6\") " Sep 07 16:32:54.105409 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.105165 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lzjr5\" (UniqueName: \"kubernetes.io/projected/7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6-kube-api-access-lzjr5\") pod \"7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6\" (UID: \"7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6\") " Sep 07 16:32:54.105409 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.105199 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6-istio-podinfo\") pod \"7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6\" (UID: \"7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6\") " Sep 07 16:32:54.105409 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.105243 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6" (UID: "7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 16:32:54.105409 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.105280 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6-credential-socket\") pod \"7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6\" (UID: \"7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6\") " Sep 07 16:32:54.105409 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.105350 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6-istio-data" (OuterVolumeSpecName: "istio-data") pod "7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6" (UID: "7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 16:32:54.105409 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.105400 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6-istiod-ca-cert\") pod \"7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6\" (UID: \"7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6\") " Sep 07 16:32:54.105704 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.105458 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6-workload-certs\") pod \"7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6\" (UID: \"7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6\") " Sep 07 16:32:54.105704 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.105505 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6-istio-envoy\") pod \"7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6\" (UID: \"7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6\") " Sep 07 16:32:54.105704 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.105544 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6" (UID: "7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 16:32:54.105848 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.105761 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6" (UID: "7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 16:32:54.105848 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.105789 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6" (UID: "7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 16:32:54.106013 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.105889 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6-credential-socket\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:32:54.106013 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.105906 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6-istiod-ca-cert\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:32:54.106013 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.105919 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6-workload-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:32:54.106013 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.105931 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6-workload-socket\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:32:54.106013 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.105943 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6-istio-data\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:32:54.107645 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.107621 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6" (UID: "7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 07 16:32:54.107809 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.107792 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6-istio-token" (OuterVolumeSpecName: "istio-token") pod "7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6" (UID: "7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 16:32:54.108135 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.108106 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6" (UID: "7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 16:32:54.108135 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.108115 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6-kube-api-access-lzjr5" (OuterVolumeSpecName: "kube-api-access-lzjr5") pod "7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6" (UID: "7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6"). InnerVolumeSpecName "kube-api-access-lzjr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 16:32:54.206868 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.206835 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6-istio-envoy\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:32:54.206868 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.206866 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6-istio-token\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:32:54.206868 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.206875 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzjr5\" (UniqueName: \"kubernetes.io/projected/7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6-kube-api-access-lzjr5\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:32:54.207134 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.206884 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6-istio-podinfo\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:32:54.315953 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.315915 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-39437224-6bcf8b5dc8-zsflq"] Sep 07 16:32:54.316304 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.316291 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6" containerName="istio-proxy" Sep 07 16:32:54.316351 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.316308 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6" containerName="istio-proxy" Sep 07 16:32:54.316351 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.316332 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6" containerName="istio-proxy" Sep 07 16:32:54.316351 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.316339 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6" containerName="istio-proxy" Sep 07 16:32:54.316438 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.316399 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6" containerName="istio-proxy" Sep 07 16:32:54.316438 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.316409 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6" containerName="istio-proxy" Sep 07 16:32:54.318618 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.318604 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-39437224-6bcf8b5dc8-zsflq" Sep 07 16:32:54.320825 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.320800 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-39437224\"" Sep 07 16:32:54.320958 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.320843 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-39437224-dockercfg-5vwkp\"" Sep 07 16:32:54.333677 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.333646 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-39437224-6bcf8b5dc8-zsflq"] Sep 07 16:32:54.352822 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.352783 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-39437224-b555d789-cw989"] Sep 07 16:32:54.355374 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.355307 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-39437224-b555d789-cw989" Sep 07 16:32:54.364639 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.364604 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-39437224-b555d789-cw989"] Sep 07 16:32:54.411414 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.411320 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x59xc\" (UniqueName: \"kubernetes.io/projected/70f8b275-b472-44f6-b440-6d9ef383939f-kube-api-access-x59xc\") pod \"payload-pre-processing-e2e-derive-39437224-6bcf8b5dc8-zsflq\" (UID: \"70f8b275-b472-44f6-b440-6d9ef383939f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-39437224-6bcf8b5dc8-zsflq" Sep 07 16:32:54.411735 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.411419 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/650adc18-5822-4185-a681-e32a3a08952c-plugins-config-volume\") pod \"payload-processing-e2e-derive-39437224-b555d789-cw989\" (UID: \"650adc18-5822-4185-a681-e32a3a08952c\") " pod="openshift-ingress/payload-processing-e2e-derive-39437224-b555d789-cw989" Sep 07 16:32:54.411735 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.411463 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrcq9\" (UniqueName: \"kubernetes.io/projected/650adc18-5822-4185-a681-e32a3a08952c-kube-api-access-wrcq9\") pod \"payload-processing-e2e-derive-39437224-b555d789-cw989\" (UID: \"650adc18-5822-4185-a681-e32a3a08952c\") " pod="openshift-ingress/payload-processing-e2e-derive-39437224-b555d789-cw989" Sep 07 16:32:54.411735 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.411554 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70f8b275-b472-44f6-b440-6d9ef383939f-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-39437224-6bcf8b5dc8-zsflq\" (UID: \"70f8b275-b472-44f6-b440-6d9ef383939f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-39437224-6bcf8b5dc8-zsflq" Sep 07 16:32:54.513073 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.513016 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/650adc18-5822-4185-a681-e32a3a08952c-plugins-config-volume\") pod \"payload-processing-e2e-derive-39437224-b555d789-cw989\" (UID: \"650adc18-5822-4185-a681-e32a3a08952c\") " pod="openshift-ingress/payload-processing-e2e-derive-39437224-b555d789-cw989" Sep 07 16:32:54.513073 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.513079 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrcq9\" (UniqueName: \"kubernetes.io/projected/650adc18-5822-4185-a681-e32a3a08952c-kube-api-access-wrcq9\") pod \"payload-processing-e2e-derive-39437224-b555d789-cw989\" (UID: \"650adc18-5822-4185-a681-e32a3a08952c\") " pod="openshift-ingress/payload-processing-e2e-derive-39437224-b555d789-cw989" Sep 07 16:32:54.513340 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.513099 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70f8b275-b472-44f6-b440-6d9ef383939f-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-39437224-6bcf8b5dc8-zsflq\" (UID: \"70f8b275-b472-44f6-b440-6d9ef383939f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-39437224-6bcf8b5dc8-zsflq" Sep 07 16:32:54.513340 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.513218 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x59xc\" (UniqueName: \"kubernetes.io/projected/70f8b275-b472-44f6-b440-6d9ef383939f-kube-api-access-x59xc\") pod \"payload-pre-processing-e2e-derive-39437224-6bcf8b5dc8-zsflq\" (UID: \"70f8b275-b472-44f6-b440-6d9ef383939f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-39437224-6bcf8b5dc8-zsflq" Sep 07 16:32:54.513865 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.513841 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/650adc18-5822-4185-a681-e32a3a08952c-plugins-config-volume\") pod \"payload-processing-e2e-derive-39437224-b555d789-cw989\" (UID: \"650adc18-5822-4185-a681-e32a3a08952c\") " pod="openshift-ingress/payload-processing-e2e-derive-39437224-b555d789-cw989" Sep 07 16:32:54.513908 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.513853 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70f8b275-b472-44f6-b440-6d9ef383939f-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-39437224-6bcf8b5dc8-zsflq\" (UID: \"70f8b275-b472-44f6-b440-6d9ef383939f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-39437224-6bcf8b5dc8-zsflq" Sep 07 16:32:54.520294 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.520265 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x59xc\" (UniqueName: \"kubernetes.io/projected/70f8b275-b472-44f6-b440-6d9ef383939f-kube-api-access-x59xc\") pod \"payload-pre-processing-e2e-derive-39437224-6bcf8b5dc8-zsflq\" (UID: \"70f8b275-b472-44f6-b440-6d9ef383939f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-39437224-6bcf8b5dc8-zsflq" Sep 07 16:32:54.520391 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.520338 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrcq9\" (UniqueName: \"kubernetes.io/projected/650adc18-5822-4185-a681-e32a3a08952c-kube-api-access-wrcq9\") pod \"payload-processing-e2e-derive-39437224-b555d789-cw989\" (UID: \"650adc18-5822-4185-a681-e32a3a08952c\") " pod="openshift-ingress/payload-processing-e2e-derive-39437224-b555d789-cw989" Sep 07 16:32:54.628939 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.628857 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-39437224-6bcf8b5dc8-zsflq" Sep 07 16:32:54.666573 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.666513 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-39437224-b555d789-cw989" Sep 07 16:32:54.854923 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.854883 2580 generic.go:356] "Generic (PLEG): container finished" podID="7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6" containerID="9739a13c160f480593f5c8549a3036eab1dab58f26f30d32c1abdca155d1c61a" exitCode=0 Sep 07 16:32:54.855152 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.854960 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6" event={"ID":"7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6","Type":"ContainerDied","Data":"9739a13c160f480593f5c8549a3036eab1dab58f26f30d32c1abdca155d1c61a"} Sep 07 16:32:54.855152 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.855011 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6" event={"ID":"7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6","Type":"ContainerDied","Data":"40d6335bfbb0bfeff57cec147088d00d8dbf15eb58399ab5d2f0d75383062836"} Sep 07 16:32:54.855152 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.855063 2580 scope.go:117] "RemoveContainer" containerID="9739a13c160f480593f5c8549a3036eab1dab58f26f30d32c1abdca155d1c61a" Sep 07 16:32:54.855448 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.855394 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6" Sep 07 16:32:54.865070 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.865034 2580 scope.go:117] "RemoveContainer" containerID="edec555a37ef75ecb641b3af59c808f19bf5bd15cfa1ea0a3e8df2c15bc1004e" Sep 07 16:32:54.875838 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.875814 2580 scope.go:117] "RemoveContainer" containerID="9739a13c160f480593f5c8549a3036eab1dab58f26f30d32c1abdca155d1c61a" Sep 07 16:32:54.876223 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:32:54.876201 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9739a13c160f480593f5c8549a3036eab1dab58f26f30d32c1abdca155d1c61a\": container with ID starting with 9739a13c160f480593f5c8549a3036eab1dab58f26f30d32c1abdca155d1c61a not found: ID does not exist" containerID="9739a13c160f480593f5c8549a3036eab1dab58f26f30d32c1abdca155d1c61a" Sep 07 16:32:54.876308 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.876235 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9739a13c160f480593f5c8549a3036eab1dab58f26f30d32c1abdca155d1c61a"} err="failed to get container status \"9739a13c160f480593f5c8549a3036eab1dab58f26f30d32c1abdca155d1c61a\": rpc error: code = NotFound desc = could not find container \"9739a13c160f480593f5c8549a3036eab1dab58f26f30d32c1abdca155d1c61a\": container with ID starting with 9739a13c160f480593f5c8549a3036eab1dab58f26f30d32c1abdca155d1c61a not found: ID does not exist" Sep 07 16:32:54.876308 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.876259 2580 scope.go:117] "RemoveContainer" containerID="edec555a37ef75ecb641b3af59c808f19bf5bd15cfa1ea0a3e8df2c15bc1004e" Sep 07 16:32:54.876577 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:32:54.876556 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edec555a37ef75ecb641b3af59c808f19bf5bd15cfa1ea0a3e8df2c15bc1004e\": container with ID starting with edec555a37ef75ecb641b3af59c808f19bf5bd15cfa1ea0a3e8df2c15bc1004e not found: ID does not exist" containerID="edec555a37ef75ecb641b3af59c808f19bf5bd15cfa1ea0a3e8df2c15bc1004e" Sep 07 16:32:54.876621 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.876588 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edec555a37ef75ecb641b3af59c808f19bf5bd15cfa1ea0a3e8df2c15bc1004e"} err="failed to get container status \"edec555a37ef75ecb641b3af59c808f19bf5bd15cfa1ea0a3e8df2c15bc1004e\": rpc error: code = NotFound desc = could not find container \"edec555a37ef75ecb641b3af59c808f19bf5bd15cfa1ea0a3e8df2c15bc1004e\": container with ID starting with edec555a37ef75ecb641b3af59c808f19bf5bd15cfa1ea0a3e8df2c15bc1004e not found: ID does not exist" Sep 07 16:32:54.877156 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.877141 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6"] Sep 07 16:32:54.882726 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.882666 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-b1b87a00-openshift-default-754c75444b-hrhg6"] Sep 07 16:32:54.899836 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.899816 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-39437224-6bcf8b5dc8-zsflq"] Sep 07 16:32:54.901953 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:32:54.901927 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70f8b275_b472_44f6_b440_6d9ef383939f.slice/crio-284decf3717d731f5ac723ac6ab921a7abe9d31956cbcf47a56b3d4958026ac5 WatchSource:0}: Error finding container 284decf3717d731f5ac723ac6ab921a7abe9d31956cbcf47a56b3d4958026ac5: Status 404 returned error can't find the container with id 284decf3717d731f5ac723ac6ab921a7abe9d31956cbcf47a56b3d4958026ac5 Sep 07 16:32:54.945744 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:54.945685 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-39437224-b555d789-cw989"] Sep 07 16:32:54.959065 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:32:54.959009 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod650adc18_5822_4185_a681_e32a3a08952c.slice/crio-23838e737765aa257841cf184f8f226f39e6b9893801f56ec4ae2102e438f25e WatchSource:0}: Error finding container 23838e737765aa257841cf184f8f226f39e6b9893801f56ec4ae2102e438f25e: Status 404 returned error can't find the container with id 23838e737765aa257841cf184f8f226f39e6b9893801f56ec4ae2102e438f25e Sep 07 16:32:55.418973 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:55.418938 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6" path="/var/lib/kubelet/pods/7f896d0a-c7d9-49f0-b3be-a8a0b2e752a6/volumes" Sep 07 16:32:55.862060 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:55.862011 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-39437224-6bcf8b5dc8-zsflq" event={"ID":"70f8b275-b472-44f6-b440-6d9ef383939f","Type":"ContainerStarted","Data":"e56f21ce699de88679c91b4d432489c189ad467eb432af3131c33389c9b48ce2"} Sep 07 16:32:55.862236 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:55.862072 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-39437224-6bcf8b5dc8-zsflq" event={"ID":"70f8b275-b472-44f6-b440-6d9ef383939f","Type":"ContainerStarted","Data":"284decf3717d731f5ac723ac6ab921a7abe9d31956cbcf47a56b3d4958026ac5"} Sep 07 16:32:55.862236 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:55.862136 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-39437224-6bcf8b5dc8-zsflq" Sep 07 16:32:55.863754 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:55.863731 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-39437224-b555d789-cw989" event={"ID":"650adc18-5822-4185-a681-e32a3a08952c","Type":"ContainerStarted","Data":"5b6bffac37eaf056026472fbc0b7b07d519de2143a41e03027bc079bf2a5d820"} Sep 07 16:32:55.863813 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:55.863770 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-39437224-b555d789-cw989" event={"ID":"650adc18-5822-4185-a681-e32a3a08952c","Type":"ContainerStarted","Data":"23838e737765aa257841cf184f8f226f39e6b9893801f56ec4ae2102e438f25e"} Sep 07 16:32:55.863868 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:55.863851 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-39437224-b555d789-cw989" Sep 07 16:32:55.876669 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:55.876610 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-39437224-6bcf8b5dc8-zsflq" podStartSLOduration=1.8765944399999999 podStartE2EDuration="1.87659444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:32:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:32:55.874593473 +0000 UTC m=+804.909816561" watchObservedRunningTime="2026-09-07 16:32:55.87659444 +0000 UTC m=+804.911817509" Sep 07 16:32:55.889458 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:55.889402 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-39437224-b555d789-cw989" podStartSLOduration=1.889384947 podStartE2EDuration="1.889384947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:32:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:32:55.887809202 +0000 UTC m=+804.923032303" watchObservedRunningTime="2026-09-07 16:32:55.889384947 +0000 UTC m=+804.924608016" Sep 07 16:32:57.084867 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:57.084831 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-8a3403-6bd57dc687-wbk58"] Sep 07 16:32:57.085731 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:57.085695 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-8a3403-6bd57dc687-wbk58" podUID="8e046aaf-52c5-453b-bafc-314c93f70f87" containerName="payload-processing" containerID="cri-o://4b02a669c1c4e252daaaedcc166dba7c0670940c01b60a79d4c8cd652b9f8e72" gracePeriod=30 Sep 07 16:32:57.247228 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:57.247194 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8a3403-k6xdr"] Sep 07 16:32:57.251911 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:57.251877 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8a3403-k6xdr"] Sep 07 16:32:57.357208 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:57.357015 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-8a3403-6bd57dc687-wbk58" Sep 07 16:32:57.418539 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:57.418508 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4837f09b-32a8-4661-937d-323369bcbb03" path="/var/lib/kubelet/pods/4837f09b-32a8-4661-937d-323369bcbb03/volumes" Sep 07 16:32:57.441695 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:57.441660 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e046aaf-52c5-453b-bafc-314c93f70f87-plugins-config-volume\") pod \"8e046aaf-52c5-453b-bafc-314c93f70f87\" (UID: \"8e046aaf-52c5-453b-bafc-314c93f70f87\") " Sep 07 16:32:57.441878 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:57.441784 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f6mh6\" (UniqueName: \"kubernetes.io/projected/8e046aaf-52c5-453b-bafc-314c93f70f87-kube-api-access-f6mh6\") pod \"8e046aaf-52c5-453b-bafc-314c93f70f87\" (UID: \"8e046aaf-52c5-453b-bafc-314c93f70f87\") " Sep 07 16:32:57.442186 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:57.442158 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e046aaf-52c5-453b-bafc-314c93f70f87-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8e046aaf-52c5-453b-bafc-314c93f70f87" (UID: "8e046aaf-52c5-453b-bafc-314c93f70f87"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 16:32:57.444101 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:57.444069 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e046aaf-52c5-453b-bafc-314c93f70f87-kube-api-access-f6mh6" (OuterVolumeSpecName: "kube-api-access-f6mh6") pod "8e046aaf-52c5-453b-bafc-314c93f70f87" (UID: "8e046aaf-52c5-453b-bafc-314c93f70f87"). InnerVolumeSpecName "kube-api-access-f6mh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 16:32:57.543335 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:57.543124 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6mh6\" (UniqueName: \"kubernetes.io/projected/8e046aaf-52c5-453b-bafc-314c93f70f87-kube-api-access-f6mh6\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:32:57.543335 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:57.543158 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e046aaf-52c5-453b-bafc-314c93f70f87-plugins-config-volume\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:32:57.874478 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:57.874443 2580 generic.go:356] "Generic (PLEG): container finished" podID="8e046aaf-52c5-453b-bafc-314c93f70f87" containerID="4b02a669c1c4e252daaaedcc166dba7c0670940c01b60a79d4c8cd652b9f8e72" exitCode=0 Sep 07 16:32:57.874631 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:57.874490 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-8a3403-6bd57dc687-wbk58" Sep 07 16:32:57.874631 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:57.874531 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-8a3403-6bd57dc687-wbk58" event={"ID":"8e046aaf-52c5-453b-bafc-314c93f70f87","Type":"ContainerDied","Data":"4b02a669c1c4e252daaaedcc166dba7c0670940c01b60a79d4c8cd652b9f8e72"} Sep 07 16:32:57.874631 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:57.874583 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-8a3403-6bd57dc687-wbk58" event={"ID":"8e046aaf-52c5-453b-bafc-314c93f70f87","Type":"ContainerDied","Data":"8ac910f7072a13e59b4dcc37a9b066b9718f679dd752ba9c27f3ef5434282857"} Sep 07 16:32:57.874631 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:57.874605 2580 scope.go:117] "RemoveContainer" containerID="4b02a669c1c4e252daaaedcc166dba7c0670940c01b60a79d4c8cd652b9f8e72" Sep 07 16:32:57.883759 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:57.883742 2580 scope.go:117] "RemoveContainer" containerID="4b02a669c1c4e252daaaedcc166dba7c0670940c01b60a79d4c8cd652b9f8e72" Sep 07 16:32:57.884071 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:32:57.884052 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b02a669c1c4e252daaaedcc166dba7c0670940c01b60a79d4c8cd652b9f8e72\": container with ID starting with 4b02a669c1c4e252daaaedcc166dba7c0670940c01b60a79d4c8cd652b9f8e72 not found: ID does not exist" containerID="4b02a669c1c4e252daaaedcc166dba7c0670940c01b60a79d4c8cd652b9f8e72" Sep 07 16:32:57.884125 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:57.884082 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b02a669c1c4e252daaaedcc166dba7c0670940c01b60a79d4c8cd652b9f8e72"} err="failed to get container status \"4b02a669c1c4e252daaaedcc166dba7c0670940c01b60a79d4c8cd652b9f8e72\": rpc error: code = NotFound desc = could not find container \"4b02a669c1c4e252daaaedcc166dba7c0670940c01b60a79d4c8cd652b9f8e72\": container with ID starting with 4b02a669c1c4e252daaaedcc166dba7c0670940c01b60a79d4c8cd652b9f8e72 not found: ID does not exist" Sep 07 16:32:57.894492 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:57.894462 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-8a3403-6bd57dc687-wbk58"] Sep 07 16:32:57.896155 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:57.896132 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-8a3403-6bd57dc687-wbk58"] Sep 07 16:32:59.418496 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:32:59.418461 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e046aaf-52c5-453b-bafc-314c93f70f87" path="/var/lib/kubelet/pods/8e046aaf-52c5-453b-bafc-314c93f70f87/volumes" Sep 07 16:33:03.695996 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:03.695951 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv"] Sep 07 16:33:03.696529 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:03.696239 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv" podUID="69c3fe33-98c9-404c-bccf-548c1f77e9ec" containerName="istio-proxy" containerID="cri-o://34f06df237fd476ee6c93a5d35feb5c0ca0d03006cf80c003e5065ecba2091c9" gracePeriod=30 Sep 07 16:33:06.871152 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:06.871113 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-39437224-6bcf8b5dc8-zsflq" Sep 07 16:33:06.871645 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:06.871198 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-39437224-b555d789-cw989" Sep 07 16:33:08.955563 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:08.955524 2580 generic.go:356] "Generic (PLEG): container finished" podID="69c3fe33-98c9-404c-bccf-548c1f77e9ec" containerID="34f06df237fd476ee6c93a5d35feb5c0ca0d03006cf80c003e5065ecba2091c9" exitCode=0 Sep 07 16:33:08.955948 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:08.955615 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv" event={"ID":"69c3fe33-98c9-404c-bccf-548c1f77e9ec","Type":"ContainerDied","Data":"34f06df237fd476ee6c93a5d35feb5c0ca0d03006cf80c003e5065ecba2091c9"} Sep 07 16:33:08.955948 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:08.955684 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv" event={"ID":"69c3fe33-98c9-404c-bccf-548c1f77e9ec","Type":"ContainerDied","Data":"21a2a803c9f4292cce8b9ab61620f7272cd49c85661f006594d380cab52575fe"} Sep 07 16:33:08.955948 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:08.955708 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21a2a803c9f4292cce8b9ab61620f7272cd49c85661f006594d380cab52575fe" Sep 07 16:33:08.955948 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:08.955728 2580 scope.go:117] "RemoveContainer" containerID="72fe1639848a9b30d82f8423089b7e990726c8920cef18a6690012cd9de3a842" Sep 07 16:33:08.958661 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:08.958630 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv" Sep 07 16:33:09.060179 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:09.060149 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69c3fe33-98c9-404c-bccf-548c1f77e9ec-istio-podinfo\") pod \"69c3fe33-98c9-404c-bccf-548c1f77e9ec\" (UID: \"69c3fe33-98c9-404c-bccf-548c1f77e9ec\") " Sep 07 16:33:09.060420 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:09.060197 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vwsrj\" (UniqueName: \"kubernetes.io/projected/69c3fe33-98c9-404c-bccf-548c1f77e9ec-kube-api-access-vwsrj\") pod \"69c3fe33-98c9-404c-bccf-548c1f77e9ec\" (UID: \"69c3fe33-98c9-404c-bccf-548c1f77e9ec\") " Sep 07 16:33:09.060420 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:09.060226 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69c3fe33-98c9-404c-bccf-548c1f77e9ec-istiod-ca-cert\") pod \"69c3fe33-98c9-404c-bccf-548c1f77e9ec\" (UID: \"69c3fe33-98c9-404c-bccf-548c1f77e9ec\") " Sep 07 16:33:09.060420 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:09.060253 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69c3fe33-98c9-404c-bccf-548c1f77e9ec-istio-token\") pod \"69c3fe33-98c9-404c-bccf-548c1f77e9ec\" (UID: \"69c3fe33-98c9-404c-bccf-548c1f77e9ec\") " Sep 07 16:33:09.060420 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:09.060330 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69c3fe33-98c9-404c-bccf-548c1f77e9ec-credential-socket\") pod \"69c3fe33-98c9-404c-bccf-548c1f77e9ec\" (UID: \"69c3fe33-98c9-404c-bccf-548c1f77e9ec\") " Sep 07 16:33:09.060420 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:09.060362 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69c3fe33-98c9-404c-bccf-548c1f77e9ec-workload-certs\") pod \"69c3fe33-98c9-404c-bccf-548c1f77e9ec\" (UID: \"69c3fe33-98c9-404c-bccf-548c1f77e9ec\") " Sep 07 16:33:09.060420 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:09.060395 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69c3fe33-98c9-404c-bccf-548c1f77e9ec-workload-socket\") pod \"69c3fe33-98c9-404c-bccf-548c1f77e9ec\" (UID: \"69c3fe33-98c9-404c-bccf-548c1f77e9ec\") " Sep 07 16:33:09.060694 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:09.060423 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69c3fe33-98c9-404c-bccf-548c1f77e9ec-istio-envoy\") pod \"69c3fe33-98c9-404c-bccf-548c1f77e9ec\" (UID: \"69c3fe33-98c9-404c-bccf-548c1f77e9ec\") " Sep 07 16:33:09.060694 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:09.060449 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69c3fe33-98c9-404c-bccf-548c1f77e9ec-istio-data\") pod \"69c3fe33-98c9-404c-bccf-548c1f77e9ec\" (UID: \"69c3fe33-98c9-404c-bccf-548c1f77e9ec\") " Sep 07 16:33:09.060694 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:09.060553 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69c3fe33-98c9-404c-bccf-548c1f77e9ec-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "69c3fe33-98c9-404c-bccf-548c1f77e9ec" (UID: "69c3fe33-98c9-404c-bccf-548c1f77e9ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 16:33:09.060694 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:09.060641 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69c3fe33-98c9-404c-bccf-548c1f77e9ec-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "69c3fe33-98c9-404c-bccf-548c1f77e9ec" (UID: "69c3fe33-98c9-404c-bccf-548c1f77e9ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 16:33:09.060694 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:09.060683 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69c3fe33-98c9-404c-bccf-548c1f77e9ec-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "69c3fe33-98c9-404c-bccf-548c1f77e9ec" (UID: "69c3fe33-98c9-404c-bccf-548c1f77e9ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 16:33:09.060938 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:09.060824 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69c3fe33-98c9-404c-bccf-548c1f77e9ec-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "69c3fe33-98c9-404c-bccf-548c1f77e9ec" (UID: "69c3fe33-98c9-404c-bccf-548c1f77e9ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 16:33:09.060938 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:09.060830 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69c3fe33-98c9-404c-bccf-548c1f77e9ec-istiod-ca-cert\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:33:09.060938 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:09.060848 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69c3fe33-98c9-404c-bccf-548c1f77e9ec-credential-socket\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:33:09.060938 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:09.060850 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69c3fe33-98c9-404c-bccf-548c1f77e9ec-istio-data" (OuterVolumeSpecName: "istio-data") pod "69c3fe33-98c9-404c-bccf-548c1f77e9ec" (UID: "69c3fe33-98c9-404c-bccf-548c1f77e9ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 16:33:09.060938 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:09.060861 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69c3fe33-98c9-404c-bccf-548c1f77e9ec-workload-socket\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:33:09.062617 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:09.062589 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69c3fe33-98c9-404c-bccf-548c1f77e9ec-kube-api-access-vwsrj" (OuterVolumeSpecName: "kube-api-access-vwsrj") pod "69c3fe33-98c9-404c-bccf-548c1f77e9ec" (UID: "69c3fe33-98c9-404c-bccf-548c1f77e9ec"). InnerVolumeSpecName "kube-api-access-vwsrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 16:33:09.062873 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:09.062854 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69c3fe33-98c9-404c-bccf-548c1f77e9ec-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "69c3fe33-98c9-404c-bccf-548c1f77e9ec" (UID: "69c3fe33-98c9-404c-bccf-548c1f77e9ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 16:33:09.063157 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:09.063140 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69c3fe33-98c9-404c-bccf-548c1f77e9ec-istio-token" (OuterVolumeSpecName: "istio-token") pod "69c3fe33-98c9-404c-bccf-548c1f77e9ec" (UID: "69c3fe33-98c9-404c-bccf-548c1f77e9ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 16:33:09.063219 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:09.063152 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69c3fe33-98c9-404c-bccf-548c1f77e9ec-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "69c3fe33-98c9-404c-bccf-548c1f77e9ec" (UID: "69c3fe33-98c9-404c-bccf-548c1f77e9ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 07 16:33:09.161344 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:09.161310 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69c3fe33-98c9-404c-bccf-548c1f77e9ec-istio-envoy\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:33:09.161344 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:09.161338 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69c3fe33-98c9-404c-bccf-548c1f77e9ec-istio-data\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:33:09.161344 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:09.161348 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69c3fe33-98c9-404c-bccf-548c1f77e9ec-istio-podinfo\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:33:09.161344 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:09.161357 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwsrj\" (UniqueName: \"kubernetes.io/projected/69c3fe33-98c9-404c-bccf-548c1f77e9ec-kube-api-access-vwsrj\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:33:09.161600 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:09.161366 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69c3fe33-98c9-404c-bccf-548c1f77e9ec-istio-token\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:33:09.161600 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:09.161376 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69c3fe33-98c9-404c-bccf-548c1f77e9ec-workload-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:33:09.355753 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:09.355670 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-cc0983-7978788689-9p9gm"] Sep 07 16:33:09.356044 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:09.356031 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69c3fe33-98c9-404c-bccf-548c1f77e9ec" containerName="istio-proxy" Sep 07 16:33:09.356089 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:09.356045 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="69c3fe33-98c9-404c-bccf-548c1f77e9ec" containerName="istio-proxy" Sep 07 16:33:09.356089 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:09.356054 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e046aaf-52c5-453b-bafc-314c93f70f87" containerName="payload-processing" Sep 07 16:33:09.356089 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:09.356061 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e046aaf-52c5-453b-bafc-314c93f70f87" containerName="payload-processing" Sep 07 16:33:09.356089 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:09.356067 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69c3fe33-98c9-404c-bccf-548c1f77e9ec" containerName="istio-proxy" Sep 07 16:33:09.356089 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:09.356074 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="69c3fe33-98c9-404c-bccf-548c1f77e9ec" containerName="istio-proxy" Sep 07 16:33:09.356255 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:09.356123 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="69c3fe33-98c9-404c-bccf-548c1f77e9ec" containerName="istio-proxy" Sep 07 16:33:09.356255 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:09.356133 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e046aaf-52c5-453b-bafc-314c93f70f87" containerName="payload-processing" Sep 07 16:33:09.356255 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:09.356139 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="69c3fe33-98c9-404c-bccf-548c1f77e9ec" containerName="istio-proxy" Sep 07 16:33:09.360517 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:09.360499 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-cc0983-7978788689-9p9gm" Sep 07 16:33:09.362510 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:09.362488 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-cc0983-dockercfg-b4bls\"" Sep 07 16:33:09.362510 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:09.362492 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-cc0983\"" Sep 07 16:33:09.370418 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:09.370394 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-cc0983-7978788689-9p9gm"] Sep 07 16:33:09.464250 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:09.464207 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/084f00fe-6d3f-460c-846d-8a9751ec216d-plugins-config-volume\") pod \"payload-processing-e2e-explicit-cc0983-7978788689-9p9gm\" (UID: \"084f00fe-6d3f-460c-846d-8a9751ec216d\") " pod="openshift-ingress/payload-processing-e2e-explicit-cc0983-7978788689-9p9gm" Sep 07 16:33:09.464449 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:09.464263 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5972\" (UniqueName: \"kubernetes.io/projected/084f00fe-6d3f-460c-846d-8a9751ec216d-kube-api-access-v5972\") pod \"payload-processing-e2e-explicit-cc0983-7978788689-9p9gm\" (UID: \"084f00fe-6d3f-460c-846d-8a9751ec216d\") " pod="openshift-ingress/payload-processing-e2e-explicit-cc0983-7978788689-9p9gm" Sep 07 16:33:09.565748 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:09.565707 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/084f00fe-6d3f-460c-846d-8a9751ec216d-plugins-config-volume\") pod \"payload-processing-e2e-explicit-cc0983-7978788689-9p9gm\" (UID: \"084f00fe-6d3f-460c-846d-8a9751ec216d\") " pod="openshift-ingress/payload-processing-e2e-explicit-cc0983-7978788689-9p9gm" Sep 07 16:33:09.565915 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:09.565779 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5972\" (UniqueName: \"kubernetes.io/projected/084f00fe-6d3f-460c-846d-8a9751ec216d-kube-api-access-v5972\") pod \"payload-processing-e2e-explicit-cc0983-7978788689-9p9gm\" (UID: \"084f00fe-6d3f-460c-846d-8a9751ec216d\") " pod="openshift-ingress/payload-processing-e2e-explicit-cc0983-7978788689-9p9gm" Sep 07 16:33:09.566663 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:09.566640 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/084f00fe-6d3f-460c-846d-8a9751ec216d-plugins-config-volume\") pod \"payload-processing-e2e-explicit-cc0983-7978788689-9p9gm\" (UID: \"084f00fe-6d3f-460c-846d-8a9751ec216d\") " pod="openshift-ingress/payload-processing-e2e-explicit-cc0983-7978788689-9p9gm" Sep 07 16:33:09.573968 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:09.573942 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5972\" (UniqueName: \"kubernetes.io/projected/084f00fe-6d3f-460c-846d-8a9751ec216d-kube-api-access-v5972\") pod \"payload-processing-e2e-explicit-cc0983-7978788689-9p9gm\" (UID: \"084f00fe-6d3f-460c-846d-8a9751ec216d\") " pod="openshift-ingress/payload-processing-e2e-explicit-cc0983-7978788689-9p9gm" Sep 07 16:33:09.673511 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:09.673424 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-cc0983-7978788689-9p9gm" Sep 07 16:33:09.946756 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:09.946597 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-cc0983-7978788689-9p9gm"] Sep 07 16:33:09.949239 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:33:09.949202 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod084f00fe_6d3f_460c_846d_8a9751ec216d.slice/crio-1c081532dfe9c124e1a6a3466cd77a9ec078068660f97aee93894220b4cd33cc WatchSource:0}: Error finding container 1c081532dfe9c124e1a6a3466cd77a9ec078068660f97aee93894220b4cd33cc: Status 404 returned error can't find the container with id 1c081532dfe9c124e1a6a3466cd77a9ec078068660f97aee93894220b4cd33cc Sep 07 16:33:09.962320 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:09.962288 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-cc0983-7978788689-9p9gm" event={"ID":"084f00fe-6d3f-460c-846d-8a9751ec216d","Type":"ContainerStarted","Data":"1c081532dfe9c124e1a6a3466cd77a9ec078068660f97aee93894220b4cd33cc"} Sep 07 16:33:09.963791 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:09.963770 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv" Sep 07 16:33:10.009744 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:10.009711 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv"] Sep 07 16:33:10.013043 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:10.013000 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-8a3403-openshift-default-74d58749fb-l4mnv"] Sep 07 16:33:10.970915 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:10.970878 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-cc0983-7978788689-9p9gm" event={"ID":"084f00fe-6d3f-460c-846d-8a9751ec216d","Type":"ContainerStarted","Data":"aa16ecd53221719f009e8ac60edddfcef280878552a68faebdb32b157b23b90a"} Sep 07 16:33:10.971435 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:10.970973 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-cc0983-7978788689-9p9gm" Sep 07 16:33:10.987165 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:10.987108 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-cc0983-7978788689-9p9gm" podStartSLOduration=1.987084308 podStartE2EDuration="1.987084308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:33:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:33:10.985140143 +0000 UTC m=+820.020363215" watchObservedRunningTime="2026-09-07 16:33:10.987084308 +0000 UTC m=+820.022307381" Sep 07 16:33:11.418419 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:11.418304 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69c3fe33-98c9-404c-bccf-548c1f77e9ec" path="/var/lib/kubelet/pods/69c3fe33-98c9-404c-bccf-548c1f77e9ec/volumes" Sep 07 16:33:11.977511 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:11.977473 2580 generic.go:356] "Generic (PLEG): container finished" podID="e124b934-cb8e-485f-8bec-4348c1307769" containerID="e183122982d767d924eac0637c007323eed070b282c4ef731db6bb4655d1e40e" exitCode=0 Sep 07 16:33:11.977989 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:11.977548 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-39437224-openshift-default-6db78586d-dpsvs" event={"ID":"e124b934-cb8e-485f-8bec-4348c1307769","Type":"ContainerDied","Data":"e183122982d767d924eac0637c007323eed070b282c4ef731db6bb4655d1e40e"} Sep 07 16:33:11.978123 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:11.977988 2580 scope.go:117] "RemoveContainer" containerID="e183122982d767d924eac0637c007323eed070b282c4ef731db6bb4655d1e40e" Sep 07 16:33:11.978996 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:11.978972 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:33:11.979080 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:11.979057 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:33:11.979118 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:11.979100 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:33:12.984306 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:12.984273 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-39437224-openshift-default-6db78586d-dpsvs" event={"ID":"e124b934-cb8e-485f-8bec-4348c1307769","Type":"ContainerStarted","Data":"fdab1b2bc2fe2eff3230e34331df43092d8c614ca5b84b00cb376fe38d719be6"} Sep 07 16:33:13.806606 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:13.806560 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-39437224-openshift-default-6db78586d-dpsvs" Sep 07 16:33:13.806878 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:13.806852 2580 patch_prober.go:28] interesting pod/e2e-derive-39437224-openshift-default-6db78586d-dpsvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Sep 07 16:33:13.806924 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:13.806910 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-39437224-openshift-default-6db78586d-dpsvs" podUID="e124b934-cb8e-485f-8bec-4348c1307769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Sep 07 16:33:14.806556 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:14.806519 2580 patch_prober.go:28] interesting pod/e2e-derive-39437224-openshift-default-6db78586d-dpsvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Sep 07 16:33:14.806965 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:14.806581 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-39437224-openshift-default-6db78586d-dpsvs" podUID="e124b934-cb8e-485f-8bec-4348c1307769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Sep 07 16:33:15.806420 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:15.806378 2580 patch_prober.go:28] interesting pod/e2e-derive-39437224-openshift-default-6db78586d-dpsvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Sep 07 16:33:15.806600 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:15.806443 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-39437224-openshift-default-6db78586d-dpsvs" podUID="e124b934-cb8e-485f-8bec-4348c1307769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Sep 07 16:33:16.806573 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:16.806509 2580 patch_prober.go:28] interesting pod/e2e-derive-39437224-openshift-default-6db78586d-dpsvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Sep 07 16:33:16.806802 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:16.806600 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-39437224-openshift-default-6db78586d-dpsvs" podUID="e124b934-cb8e-485f-8bec-4348c1307769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Sep 07 16:33:16.976753 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:16.976714 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-39437224-wr5vz"] Sep 07 16:33:16.980230 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:16.980213 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-39437224-wr5vz" Sep 07 16:33:16.982242 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:16.982216 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jpjxt\"" Sep 07 16:33:16.982366 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:16.982252 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 07 16:33:16.986292 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:16.986268 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-39437224-wr5vz"] Sep 07 16:33:17.142923 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:17.142826 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c31e0ee3-21c8-4bea-b495-98183475815a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-39437224-wr5vz\" (UID: \"c31e0ee3-21c8-4bea-b495-98183475815a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-39437224-wr5vz" Sep 07 16:33:17.243608 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:17.243566 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c31e0ee3-21c8-4bea-b495-98183475815a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-39437224-wr5vz\" (UID: \"c31e0ee3-21c8-4bea-b495-98183475815a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-39437224-wr5vz" Sep 07 16:33:17.244212 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:17.244190 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c31e0ee3-21c8-4bea-b495-98183475815a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-39437224-wr5vz\" (UID: \"c31e0ee3-21c8-4bea-b495-98183475815a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-39437224-wr5vz" Sep 07 16:33:17.290792 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:17.290765 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-39437224-wr5vz" Sep 07 16:33:17.587745 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:17.587716 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-39437224-wr5vz"] Sep 07 16:33:17.590708 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:33:17.590664 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc31e0ee3_21c8_4bea_b495_98183475815a.slice/crio-894700f3403bea15dc25787852b75e11b591bc0266681ae4a79b3e9090879983 WatchSource:0}: Error finding container 894700f3403bea15dc25787852b75e11b591bc0266681ae4a79b3e9090879983: Status 404 returned error can't find the container with id 894700f3403bea15dc25787852b75e11b591bc0266681ae4a79b3e9090879983 Sep 07 16:33:17.806319 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:17.806286 2580 patch_prober.go:28] interesting pod/e2e-derive-39437224-openshift-default-6db78586d-dpsvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Sep 07 16:33:17.806498 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:17.806351 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-39437224-openshift-default-6db78586d-dpsvs" podUID="e124b934-cb8e-485f-8bec-4348c1307769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Sep 07 16:33:18.012897 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:18.012860 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-39437224-wr5vz" event={"ID":"c31e0ee3-21c8-4bea-b495-98183475815a","Type":"ContainerStarted","Data":"894700f3403bea15dc25787852b75e11b591bc0266681ae4a79b3e9090879983"} Sep 07 16:33:18.812217 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:18.812132 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-39437224-openshift-default-6db78586d-dpsvs" Sep 07 16:33:18.812545 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:18.812513 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-39437224-openshift-default-6db78586d-dpsvs" Sep 07 16:33:18.813475 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:18.813452 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-39437224-openshift-default-6db78586d-dpsvs" Sep 07 16:33:19.018829 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:19.018797 2580 generic.go:356] "Generic (PLEG): container finished" podID="c31e0ee3-21c8-4bea-b495-98183475815a" containerID="3ba80e83186f6decfb2bd76b346ca5e89281c593a03f6a5933f77b03b27a7198" exitCode=0 Sep 07 16:33:19.019236 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:19.018894 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-39437224-wr5vz" event={"ID":"c31e0ee3-21c8-4bea-b495-98183475815a","Type":"ContainerDied","Data":"3ba80e83186f6decfb2bd76b346ca5e89281c593a03f6a5933f77b03b27a7198"} Sep 07 16:33:20.154965 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:20.154941 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-39437224-wr5vz" Sep 07 16:33:20.166535 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:20.166506 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c31e0ee3-21c8-4bea-b495-98183475815a-maas-api-service-ca\") pod \"c31e0ee3-21c8-4bea-b495-98183475815a\" (UID: \"c31e0ee3-21c8-4bea-b495-98183475815a\") " Sep 07 16:33:20.166864 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:20.166842 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c31e0ee3-21c8-4bea-b495-98183475815a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c31e0ee3-21c8-4bea-b495-98183475815a" (UID: "c31e0ee3-21c8-4bea-b495-98183475815a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 16:33:20.267924 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:20.267884 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c31e0ee3-21c8-4bea-b495-98183475815a-maas-api-service-ca\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:33:21.030253 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:21.030215 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-39437224-wr5vz" Sep 07 16:33:21.030500 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:21.030250 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-39437224-wr5vz" event={"ID":"c31e0ee3-21c8-4bea-b495-98183475815a","Type":"ContainerDied","Data":"894700f3403bea15dc25787852b75e11b591bc0266681ae4a79b3e9090879983"} Sep 07 16:33:21.030500 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:21.030301 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="894700f3403bea15dc25787852b75e11b591bc0266681ae4a79b3e9090879983" Sep 07 16:33:21.979967 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:21.979927 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-cc0983-7978788689-9p9gm" Sep 07 16:33:23.902004 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:23.901971 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-39437224-b555d789-cw989"] Sep 07 16:33:23.902438 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:23.902202 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-39437224-b555d789-cw989" podUID="650adc18-5822-4185-a681-e32a3a08952c" containerName="payload-processing" containerID="cri-o://5b6bffac37eaf056026472fbc0b7b07d519de2143a41e03027bc079bf2a5d820" gracePeriod=30 Sep 07 16:33:23.911276 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:23.910973 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-39437224-6bcf8b5dc8-zsflq"] Sep 07 16:33:23.911352 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:23.911316 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-39437224-6bcf8b5dc8-zsflq" podUID="70f8b275-b472-44f6-b440-6d9ef383939f" containerName="payload-pre-processing" containerID="cri-o://e56f21ce699de88679c91b4d432489c189ad467eb432af3131c33389c9b48ce2" gracePeriod=30 Sep 07 16:33:24.047913 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:24.047879 2580 generic.go:356] "Generic (PLEG): container finished" podID="70f8b275-b472-44f6-b440-6d9ef383939f" containerID="e56f21ce699de88679c91b4d432489c189ad467eb432af3131c33389c9b48ce2" exitCode=0 Sep 07 16:33:24.048177 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:24.047957 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-39437224-6bcf8b5dc8-zsflq" event={"ID":"70f8b275-b472-44f6-b440-6d9ef383939f","Type":"ContainerDied","Data":"e56f21ce699de88679c91b4d432489c189ad467eb432af3131c33389c9b48ce2"} Sep 07 16:33:24.049834 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:24.049807 2580 generic.go:356] "Generic (PLEG): container finished" podID="650adc18-5822-4185-a681-e32a3a08952c" containerID="5b6bffac37eaf056026472fbc0b7b07d519de2143a41e03027bc079bf2a5d820" exitCode=0 Sep 07 16:33:24.049976 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:24.049888 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-39437224-b555d789-cw989" event={"ID":"650adc18-5822-4185-a681-e32a3a08952c","Type":"ContainerDied","Data":"5b6bffac37eaf056026472fbc0b7b07d519de2143a41e03027bc079bf2a5d820"} Sep 07 16:33:24.057006 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:24.056979 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-39437224-wr5vz"] Sep 07 16:33:24.060073 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:24.060051 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-39437224-wr5vz"] Sep 07 16:33:24.175000 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:24.174841 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-39437224-openshift-default-6db78586d-dpsvs"] Sep 07 16:33:24.175918 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:24.175773 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-39437224-openshift-default-6db78586d-dpsvs" podUID="e124b934-cb8e-485f-8bec-4348c1307769" containerName="istio-proxy" containerID="cri-o://fdab1b2bc2fe2eff3230e34331df43092d8c614ca5b84b00cb376fe38d719be6" gracePeriod=30 Sep 07 16:33:24.215126 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:24.215093 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-39437224-b555d789-cw989" Sep 07 16:33:24.218439 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:24.218412 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-39437224-6bcf8b5dc8-zsflq" Sep 07 16:33:24.302523 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:24.302488 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrcq9\" (UniqueName: \"kubernetes.io/projected/650adc18-5822-4185-a681-e32a3a08952c-kube-api-access-wrcq9\") pod \"650adc18-5822-4185-a681-e32a3a08952c\" (UID: \"650adc18-5822-4185-a681-e32a3a08952c\") " Sep 07 16:33:24.302715 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:24.302593 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70f8b275-b472-44f6-b440-6d9ef383939f-plugins-config-volume\") pod \"70f8b275-b472-44f6-b440-6d9ef383939f\" (UID: \"70f8b275-b472-44f6-b440-6d9ef383939f\") " Sep 07 16:33:24.302715 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:24.302642 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x59xc\" (UniqueName: \"kubernetes.io/projected/70f8b275-b472-44f6-b440-6d9ef383939f-kube-api-access-x59xc\") pod \"70f8b275-b472-44f6-b440-6d9ef383939f\" (UID: \"70f8b275-b472-44f6-b440-6d9ef383939f\") " Sep 07 16:33:24.302715 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:24.302675 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/650adc18-5822-4185-a681-e32a3a08952c-plugins-config-volume\") pod \"650adc18-5822-4185-a681-e32a3a08952c\" (UID: \"650adc18-5822-4185-a681-e32a3a08952c\") " Sep 07 16:33:24.303067 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:24.303040 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70f8b275-b472-44f6-b440-6d9ef383939f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "70f8b275-b472-44f6-b440-6d9ef383939f" (UID: "70f8b275-b472-44f6-b440-6d9ef383939f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 16:33:24.303176 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:24.303154 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/650adc18-5822-4185-a681-e32a3a08952c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "650adc18-5822-4185-a681-e32a3a08952c" (UID: "650adc18-5822-4185-a681-e32a3a08952c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 16:33:24.304793 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:24.304766 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/650adc18-5822-4185-a681-e32a3a08952c-kube-api-access-wrcq9" (OuterVolumeSpecName: "kube-api-access-wrcq9") pod "650adc18-5822-4185-a681-e32a3a08952c" (UID: "650adc18-5822-4185-a681-e32a3a08952c"). InnerVolumeSpecName "kube-api-access-wrcq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 16:33:24.304887 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:24.304857 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70f8b275-b472-44f6-b440-6d9ef383939f-kube-api-access-x59xc" (OuterVolumeSpecName: "kube-api-access-x59xc") pod "70f8b275-b472-44f6-b440-6d9ef383939f" (UID: "70f8b275-b472-44f6-b440-6d9ef383939f"). InnerVolumeSpecName "kube-api-access-x59xc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 16:33:24.403346 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:24.403312 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70f8b275-b472-44f6-b440-6d9ef383939f-plugins-config-volume\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:33:24.403346 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:24.403341 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x59xc\" (UniqueName: \"kubernetes.io/projected/70f8b275-b472-44f6-b440-6d9ef383939f-kube-api-access-x59xc\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:33:24.403346 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:24.403350 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/650adc18-5822-4185-a681-e32a3a08952c-plugins-config-volume\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:33:24.403346 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:24.403359 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrcq9\" (UniqueName: \"kubernetes.io/projected/650adc18-5822-4185-a681-e32a3a08952c-kube-api-access-wrcq9\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:33:25.056220 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:25.056188 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-39437224-6bcf8b5dc8-zsflq" Sep 07 16:33:25.056692 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:25.056184 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-39437224-6bcf8b5dc8-zsflq" event={"ID":"70f8b275-b472-44f6-b440-6d9ef383939f","Type":"ContainerDied","Data":"284decf3717d731f5ac723ac6ab921a7abe9d31956cbcf47a56b3d4958026ac5"} Sep 07 16:33:25.056692 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:25.056335 2580 scope.go:117] "RemoveContainer" containerID="e56f21ce699de88679c91b4d432489c189ad467eb432af3131c33389c9b48ce2" Sep 07 16:33:25.057780 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:25.057695 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-39437224-b555d789-cw989" Sep 07 16:33:25.057780 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:25.057688 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-39437224-b555d789-cw989" event={"ID":"650adc18-5822-4185-a681-e32a3a08952c","Type":"ContainerDied","Data":"23838e737765aa257841cf184f8f226f39e6b9893801f56ec4ae2102e438f25e"} Sep 07 16:33:25.067015 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:25.066995 2580 scope.go:117] "RemoveContainer" containerID="5b6bffac37eaf056026472fbc0b7b07d519de2143a41e03027bc079bf2a5d820" Sep 07 16:33:25.083714 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:25.083689 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-39437224-b555d789-cw989"] Sep 07 16:33:25.086097 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:25.086073 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-39437224-b555d789-cw989"] Sep 07 16:33:25.095134 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:25.095111 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-39437224-6bcf8b5dc8-zsflq"] Sep 07 16:33:25.100755 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:25.100732 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-39437224-6bcf8b5dc8-zsflq"] Sep 07 16:33:25.418013 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:25.417928 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="650adc18-5822-4185-a681-e32a3a08952c" path="/var/lib/kubelet/pods/650adc18-5822-4185-a681-e32a3a08952c/volumes" Sep 07 16:33:25.418315 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:25.418301 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70f8b275-b472-44f6-b440-6d9ef383939f" path="/var/lib/kubelet/pods/70f8b275-b472-44f6-b440-6d9ef383939f/volumes" Sep 07 16:33:25.418674 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:25.418659 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c31e0ee3-21c8-4bea-b495-98183475815a" path="/var/lib/kubelet/pods/c31e0ee3-21c8-4bea-b495-98183475815a/volumes" Sep 07 16:33:29.427761 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:29.427734 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-39437224-openshift-default-6db78586d-dpsvs" Sep 07 16:33:29.448753 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:29.448718 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e124b934-cb8e-485f-8bec-4348c1307769-credential-socket\") pod \"e124b934-cb8e-485f-8bec-4348c1307769\" (UID: \"e124b934-cb8e-485f-8bec-4348c1307769\") " Sep 07 16:33:29.448753 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:29.448754 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjx8v\" (UniqueName: \"kubernetes.io/projected/e124b934-cb8e-485f-8bec-4348c1307769-kube-api-access-jjx8v\") pod \"e124b934-cb8e-485f-8bec-4348c1307769\" (UID: \"e124b934-cb8e-485f-8bec-4348c1307769\") " Sep 07 16:33:29.448996 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:29.448789 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e124b934-cb8e-485f-8bec-4348c1307769-istio-envoy\") pod \"e124b934-cb8e-485f-8bec-4348c1307769\" (UID: \"e124b934-cb8e-485f-8bec-4348c1307769\") " Sep 07 16:33:29.448996 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:29.448805 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e124b934-cb8e-485f-8bec-4348c1307769-istio-data\") pod \"e124b934-cb8e-485f-8bec-4348c1307769\" (UID: \"e124b934-cb8e-485f-8bec-4348c1307769\") " Sep 07 16:33:29.448996 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:29.448836 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e124b934-cb8e-485f-8bec-4348c1307769-istiod-ca-cert\") pod \"e124b934-cb8e-485f-8bec-4348c1307769\" (UID: \"e124b934-cb8e-485f-8bec-4348c1307769\") " Sep 07 16:33:29.448996 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:29.448897 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e124b934-cb8e-485f-8bec-4348c1307769-workload-certs\") pod \"e124b934-cb8e-485f-8bec-4348c1307769\" (UID: \"e124b934-cb8e-485f-8bec-4348c1307769\") " Sep 07 16:33:29.448996 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:29.448945 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e124b934-cb8e-485f-8bec-4348c1307769-istio-token\") pod \"e124b934-cb8e-485f-8bec-4348c1307769\" (UID: \"e124b934-cb8e-485f-8bec-4348c1307769\") " Sep 07 16:33:29.448996 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:29.448976 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e124b934-cb8e-485f-8bec-4348c1307769-istio-podinfo\") pod \"e124b934-cb8e-485f-8bec-4348c1307769\" (UID: \"e124b934-cb8e-485f-8bec-4348c1307769\") " Sep 07 16:33:29.449325 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:29.449004 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e124b934-cb8e-485f-8bec-4348c1307769-workload-socket\") pod \"e124b934-cb8e-485f-8bec-4348c1307769\" (UID: \"e124b934-cb8e-485f-8bec-4348c1307769\") " Sep 07 16:33:29.449448 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:29.448997 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e124b934-cb8e-485f-8bec-4348c1307769-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e124b934-cb8e-485f-8bec-4348c1307769" (UID: "e124b934-cb8e-485f-8bec-4348c1307769"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 16:33:29.449515 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:29.449241 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e124b934-cb8e-485f-8bec-4348c1307769-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e124b934-cb8e-485f-8bec-4348c1307769" (UID: "e124b934-cb8e-485f-8bec-4348c1307769"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 16:33:29.449515 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:29.449342 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e124b934-cb8e-485f-8bec-4348c1307769-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e124b934-cb8e-485f-8bec-4348c1307769" (UID: "e124b934-cb8e-485f-8bec-4348c1307769"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 16:33:29.449515 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:29.449420 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e124b934-cb8e-485f-8bec-4348c1307769-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e124b934-cb8e-485f-8bec-4348c1307769" (UID: "e124b934-cb8e-485f-8bec-4348c1307769"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 16:33:29.449873 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:29.449824 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e124b934-cb8e-485f-8bec-4348c1307769-istio-data" (OuterVolumeSpecName: "istio-data") pod "e124b934-cb8e-485f-8bec-4348c1307769" (UID: "e124b934-cb8e-485f-8bec-4348c1307769"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 16:33:29.451472 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:29.451446 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e124b934-cb8e-485f-8bec-4348c1307769-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e124b934-cb8e-485f-8bec-4348c1307769" (UID: "e124b934-cb8e-485f-8bec-4348c1307769"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 16:33:29.451961 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:29.451929 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e124b934-cb8e-485f-8bec-4348c1307769-istio-token" (OuterVolumeSpecName: "istio-token") pod "e124b934-cb8e-485f-8bec-4348c1307769" (UID: "e124b934-cb8e-485f-8bec-4348c1307769"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 16:33:29.452108 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:29.451929 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e124b934-cb8e-485f-8bec-4348c1307769-kube-api-access-jjx8v" (OuterVolumeSpecName: "kube-api-access-jjx8v") pod "e124b934-cb8e-485f-8bec-4348c1307769" (UID: "e124b934-cb8e-485f-8bec-4348c1307769"). InnerVolumeSpecName "kube-api-access-jjx8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 16:33:29.452108 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:29.452016 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e124b934-cb8e-485f-8bec-4348c1307769-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e124b934-cb8e-485f-8bec-4348c1307769" (UID: "e124b934-cb8e-485f-8bec-4348c1307769"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 07 16:33:29.550722 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:29.550630 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e124b934-cb8e-485f-8bec-4348c1307769-istiod-ca-cert\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:33:29.550722 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:29.550665 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e124b934-cb8e-485f-8bec-4348c1307769-workload-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:33:29.550722 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:29.550674 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e124b934-cb8e-485f-8bec-4348c1307769-istio-token\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:33:29.550722 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:29.550682 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e124b934-cb8e-485f-8bec-4348c1307769-istio-podinfo\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:33:29.550722 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:29.550690 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e124b934-cb8e-485f-8bec-4348c1307769-workload-socket\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:33:29.550722 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:29.550699 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e124b934-cb8e-485f-8bec-4348c1307769-credential-socket\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:33:29.550722 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:29.550708 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjx8v\" (UniqueName: \"kubernetes.io/projected/e124b934-cb8e-485f-8bec-4348c1307769-kube-api-access-jjx8v\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:33:29.550722 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:29.550716 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e124b934-cb8e-485f-8bec-4348c1307769-istio-envoy\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:33:29.550722 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:29.550724 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e124b934-cb8e-485f-8bec-4348c1307769-istio-data\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:33:30.087431 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:30.087395 2580 generic.go:356] "Generic (PLEG): container finished" podID="e124b934-cb8e-485f-8bec-4348c1307769" containerID="fdab1b2bc2fe2eff3230e34331df43092d8c614ca5b84b00cb376fe38d719be6" exitCode=0 Sep 07 16:33:30.087687 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:30.087441 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-39437224-openshift-default-6db78586d-dpsvs" Sep 07 16:33:30.087687 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:30.087482 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-39437224-openshift-default-6db78586d-dpsvs" event={"ID":"e124b934-cb8e-485f-8bec-4348c1307769","Type":"ContainerDied","Data":"fdab1b2bc2fe2eff3230e34331df43092d8c614ca5b84b00cb376fe38d719be6"} Sep 07 16:33:30.087687 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:30.087539 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-39437224-openshift-default-6db78586d-dpsvs" event={"ID":"e124b934-cb8e-485f-8bec-4348c1307769","Type":"ContainerDied","Data":"174ea7dfdf5e8427ff34f0ab3c3e4525d216d76c7ce0e01199dff153e43548af"} Sep 07 16:33:30.087687 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:30.087561 2580 scope.go:117] "RemoveContainer" containerID="fdab1b2bc2fe2eff3230e34331df43092d8c614ca5b84b00cb376fe38d719be6" Sep 07 16:33:30.102784 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:30.102765 2580 scope.go:117] "RemoveContainer" containerID="e183122982d767d924eac0637c007323eed070b282c4ef731db6bb4655d1e40e" Sep 07 16:33:30.111461 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:30.111433 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-39437224-openshift-default-6db78586d-dpsvs"] Sep 07 16:33:30.112114 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:30.112091 2580 scope.go:117] "RemoveContainer" containerID="fdab1b2bc2fe2eff3230e34331df43092d8c614ca5b84b00cb376fe38d719be6" Sep 07 16:33:30.112440 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:33:30.112418 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdab1b2bc2fe2eff3230e34331df43092d8c614ca5b84b00cb376fe38d719be6\": container with ID starting with fdab1b2bc2fe2eff3230e34331df43092d8c614ca5b84b00cb376fe38d719be6 not found: ID does not exist" containerID="fdab1b2bc2fe2eff3230e34331df43092d8c614ca5b84b00cb376fe38d719be6" Sep 07 16:33:30.112556 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:30.112484 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdab1b2bc2fe2eff3230e34331df43092d8c614ca5b84b00cb376fe38d719be6"} err="failed to get container status \"fdab1b2bc2fe2eff3230e34331df43092d8c614ca5b84b00cb376fe38d719be6\": rpc error: code = NotFound desc = could not find container \"fdab1b2bc2fe2eff3230e34331df43092d8c614ca5b84b00cb376fe38d719be6\": container with ID starting with fdab1b2bc2fe2eff3230e34331df43092d8c614ca5b84b00cb376fe38d719be6 not found: ID does not exist" Sep 07 16:33:30.112710 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:30.112560 2580 scope.go:117] "RemoveContainer" containerID="e183122982d767d924eac0637c007323eed070b282c4ef731db6bb4655d1e40e" Sep 07 16:33:30.112978 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:33:30.112945 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e183122982d767d924eac0637c007323eed070b282c4ef731db6bb4655d1e40e\": container with ID starting with e183122982d767d924eac0637c007323eed070b282c4ef731db6bb4655d1e40e not found: ID does not exist" containerID="e183122982d767d924eac0637c007323eed070b282c4ef731db6bb4655d1e40e" Sep 07 16:33:30.113134 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:30.112983 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e183122982d767d924eac0637c007323eed070b282c4ef731db6bb4655d1e40e"} err="failed to get container status \"e183122982d767d924eac0637c007323eed070b282c4ef731db6bb4655d1e40e\": rpc error: code = NotFound desc = could not find container \"e183122982d767d924eac0637c007323eed070b282c4ef731db6bb4655d1e40e\": container with ID starting with e183122982d767d924eac0637c007323eed070b282c4ef731db6bb4655d1e40e not found: ID does not exist" Sep 07 16:33:30.115069 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:30.115047 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-39437224-openshift-default-6db78586d-dpsvs"] Sep 07 16:33:31.417895 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:31.417862 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e124b934-cb8e-485f-8bec-4348c1307769" path="/var/lib/kubelet/pods/e124b934-cb8e-485f-8bec-4348c1307769/volumes" Sep 07 16:33:34.231579 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:34.231545 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-a7d399/test-model-a7d399-kserve-56f8498599-vwp4s"] Sep 07 16:33:34.231965 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:34.231861 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e124b934-cb8e-485f-8bec-4348c1307769" containerName="istio-proxy" Sep 07 16:33:34.231965 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:34.231872 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e124b934-cb8e-485f-8bec-4348c1307769" containerName="istio-proxy" Sep 07 16:33:34.231965 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:34.231881 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c31e0ee3-21c8-4bea-b495-98183475815a" containerName="revoke-keys" Sep 07 16:33:34.231965 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:34.231888 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c31e0ee3-21c8-4bea-b495-98183475815a" containerName="revoke-keys" Sep 07 16:33:34.231965 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:34.231903 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70f8b275-b472-44f6-b440-6d9ef383939f" containerName="payload-pre-processing" Sep 07 16:33:34.231965 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:34.231910 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="70f8b275-b472-44f6-b440-6d9ef383939f" containerName="payload-pre-processing" Sep 07 16:33:34.231965 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:34.231917 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="650adc18-5822-4185-a681-e32a3a08952c" containerName="payload-processing" Sep 07 16:33:34.231965 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:34.231922 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="650adc18-5822-4185-a681-e32a3a08952c" containerName="payload-processing" Sep 07 16:33:34.232253 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:34.231973 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="70f8b275-b472-44f6-b440-6d9ef383939f" containerName="payload-pre-processing" Sep 07 16:33:34.232253 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:34.231981 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e124b934-cb8e-485f-8bec-4348c1307769" containerName="istio-proxy" Sep 07 16:33:34.232253 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:34.231989 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c31e0ee3-21c8-4bea-b495-98183475815a" containerName="revoke-keys" Sep 07 16:33:34.232253 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:34.231995 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e124b934-cb8e-485f-8bec-4348c1307769" containerName="istio-proxy" Sep 07 16:33:34.232253 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:34.232000 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="650adc18-5822-4185-a681-e32a3a08952c" containerName="payload-processing" Sep 07 16:33:34.236868 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:34.236835 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a7d399/test-model-a7d399-kserve-56f8498599-vwp4s" Sep 07 16:33:34.239223 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:34.239196 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-a7d399\"/\"kube-root-ca.crt\"" Sep 07 16:33:34.239991 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:34.239972 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-a7d399\"/\"openshift-service-ca.crt\"" Sep 07 16:33:34.240122 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:34.240090 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-a7d399\"/\"default-dockercfg-7pzgl\"" Sep 07 16:33:34.240234 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:34.240212 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-a7d399\"/\"test-model-a7d399-kserve-self-signed-certs\"" Sep 07 16:33:34.247269 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:34.247245 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-a7d399/test-model-a7d399-kserve-56f8498599-vwp4s"] Sep 07 16:33:34.293290 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:34.293253 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65552f5b-548e-4df7-a581-eae11be6962e-dshm\") pod \"test-model-a7d399-kserve-56f8498599-vwp4s\" (UID: \"65552f5b-548e-4df7-a581-eae11be6962e\") " pod="ai-tenant-e2e-inf-b-a7d399/test-model-a7d399-kserve-56f8498599-vwp4s" Sep 07 16:33:34.293460 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:34.293327 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65552f5b-548e-4df7-a581-eae11be6962e-tls-certs\") pod \"test-model-a7d399-kserve-56f8498599-vwp4s\" (UID: \"65552f5b-548e-4df7-a581-eae11be6962e\") " pod="ai-tenant-e2e-inf-b-a7d399/test-model-a7d399-kserve-56f8498599-vwp4s" Sep 07 16:33:34.293460 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:34.293370 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s68c9\" (UniqueName: \"kubernetes.io/projected/65552f5b-548e-4df7-a581-eae11be6962e-kube-api-access-s68c9\") pod \"test-model-a7d399-kserve-56f8498599-vwp4s\" (UID: \"65552f5b-548e-4df7-a581-eae11be6962e\") " pod="ai-tenant-e2e-inf-b-a7d399/test-model-a7d399-kserve-56f8498599-vwp4s" Sep 07 16:33:34.293460 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:34.293394 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65552f5b-548e-4df7-a581-eae11be6962e-tmp-dir\") pod \"test-model-a7d399-kserve-56f8498599-vwp4s\" (UID: \"65552f5b-548e-4df7-a581-eae11be6962e\") " pod="ai-tenant-e2e-inf-b-a7d399/test-model-a7d399-kserve-56f8498599-vwp4s" Sep 07 16:33:34.293460 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:34.293422 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65552f5b-548e-4df7-a581-eae11be6962e-model-cache\") pod \"test-model-a7d399-kserve-56f8498599-vwp4s\" (UID: \"65552f5b-548e-4df7-a581-eae11be6962e\") " pod="ai-tenant-e2e-inf-b-a7d399/test-model-a7d399-kserve-56f8498599-vwp4s" Sep 07 16:33:34.293460 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:34.293447 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65552f5b-548e-4df7-a581-eae11be6962e-home\") pod \"test-model-a7d399-kserve-56f8498599-vwp4s\" (UID: \"65552f5b-548e-4df7-a581-eae11be6962e\") " pod="ai-tenant-e2e-inf-b-a7d399/test-model-a7d399-kserve-56f8498599-vwp4s" Sep 07 16:33:34.394919 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:34.394876 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65552f5b-548e-4df7-a581-eae11be6962e-dshm\") pod \"test-model-a7d399-kserve-56f8498599-vwp4s\" (UID: \"65552f5b-548e-4df7-a581-eae11be6962e\") " pod="ai-tenant-e2e-inf-b-a7d399/test-model-a7d399-kserve-56f8498599-vwp4s" Sep 07 16:33:34.395159 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:34.394956 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65552f5b-548e-4df7-a581-eae11be6962e-tls-certs\") pod \"test-model-a7d399-kserve-56f8498599-vwp4s\" (UID: \"65552f5b-548e-4df7-a581-eae11be6962e\") " pod="ai-tenant-e2e-inf-b-a7d399/test-model-a7d399-kserve-56f8498599-vwp4s" Sep 07 16:33:34.395159 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:34.394992 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s68c9\" (UniqueName: \"kubernetes.io/projected/65552f5b-548e-4df7-a581-eae11be6962e-kube-api-access-s68c9\") pod \"test-model-a7d399-kserve-56f8498599-vwp4s\" (UID: \"65552f5b-548e-4df7-a581-eae11be6962e\") " pod="ai-tenant-e2e-inf-b-a7d399/test-model-a7d399-kserve-56f8498599-vwp4s" Sep 07 16:33:34.395159 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:34.395048 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65552f5b-548e-4df7-a581-eae11be6962e-tmp-dir\") pod \"test-model-a7d399-kserve-56f8498599-vwp4s\" (UID: \"65552f5b-548e-4df7-a581-eae11be6962e\") " pod="ai-tenant-e2e-inf-b-a7d399/test-model-a7d399-kserve-56f8498599-vwp4s" Sep 07 16:33:34.395159 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:34.395076 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65552f5b-548e-4df7-a581-eae11be6962e-model-cache\") pod \"test-model-a7d399-kserve-56f8498599-vwp4s\" (UID: \"65552f5b-548e-4df7-a581-eae11be6962e\") " pod="ai-tenant-e2e-inf-b-a7d399/test-model-a7d399-kserve-56f8498599-vwp4s" Sep 07 16:33:34.395159 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:34.395111 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65552f5b-548e-4df7-a581-eae11be6962e-home\") pod \"test-model-a7d399-kserve-56f8498599-vwp4s\" (UID: \"65552f5b-548e-4df7-a581-eae11be6962e\") " pod="ai-tenant-e2e-inf-b-a7d399/test-model-a7d399-kserve-56f8498599-vwp4s" Sep 07 16:33:34.395493 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:34.395471 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65552f5b-548e-4df7-a581-eae11be6962e-tmp-dir\") pod \"test-model-a7d399-kserve-56f8498599-vwp4s\" (UID: \"65552f5b-548e-4df7-a581-eae11be6962e\") " pod="ai-tenant-e2e-inf-b-a7d399/test-model-a7d399-kserve-56f8498599-vwp4s" Sep 07 16:33:34.395532 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:34.395498 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65552f5b-548e-4df7-a581-eae11be6962e-model-cache\") pod \"test-model-a7d399-kserve-56f8498599-vwp4s\" (UID: \"65552f5b-548e-4df7-a581-eae11be6962e\") " pod="ai-tenant-e2e-inf-b-a7d399/test-model-a7d399-kserve-56f8498599-vwp4s" Sep 07 16:33:34.395568 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:34.395546 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65552f5b-548e-4df7-a581-eae11be6962e-home\") pod \"test-model-a7d399-kserve-56f8498599-vwp4s\" (UID: \"65552f5b-548e-4df7-a581-eae11be6962e\") " pod="ai-tenant-e2e-inf-b-a7d399/test-model-a7d399-kserve-56f8498599-vwp4s" Sep 07 16:33:34.397416 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:34.397390 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65552f5b-548e-4df7-a581-eae11be6962e-dshm\") pod \"test-model-a7d399-kserve-56f8498599-vwp4s\" (UID: \"65552f5b-548e-4df7-a581-eae11be6962e\") " pod="ai-tenant-e2e-inf-b-a7d399/test-model-a7d399-kserve-56f8498599-vwp4s" Sep 07 16:33:34.397651 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:34.397635 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65552f5b-548e-4df7-a581-eae11be6962e-tls-certs\") pod \"test-model-a7d399-kserve-56f8498599-vwp4s\" (UID: \"65552f5b-548e-4df7-a581-eae11be6962e\") " pod="ai-tenant-e2e-inf-b-a7d399/test-model-a7d399-kserve-56f8498599-vwp4s" Sep 07 16:33:34.402319 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:34.402295 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s68c9\" (UniqueName: \"kubernetes.io/projected/65552f5b-548e-4df7-a581-eae11be6962e-kube-api-access-s68c9\") pod \"test-model-a7d399-kserve-56f8498599-vwp4s\" (UID: \"65552f5b-548e-4df7-a581-eae11be6962e\") " pod="ai-tenant-e2e-inf-b-a7d399/test-model-a7d399-kserve-56f8498599-vwp4s" Sep 07 16:33:34.547084 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:34.546958 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a7d399/test-model-a7d399-kserve-56f8498599-vwp4s" Sep 07 16:33:34.819839 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:34.819635 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-a7d399/test-model-a7d399-kserve-56f8498599-vwp4s"] Sep 07 16:33:34.822544 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:33:34.822513 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65552f5b_548e_4df7_a581_eae11be6962e.slice/crio-ddd7aca7c13c165e57ba99902c30f99590708706c624323a0063421f1268938f WatchSource:0}: Error finding container ddd7aca7c13c165e57ba99902c30f99590708706c624323a0063421f1268938f: Status 404 returned error can't find the container with id ddd7aca7c13c165e57ba99902c30f99590708706c624323a0063421f1268938f Sep 07 16:33:35.115786 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:35.115682 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a7d399/test-model-a7d399-kserve-56f8498599-vwp4s" event={"ID":"65552f5b-548e-4df7-a581-eae11be6962e","Type":"ContainerStarted","Data":"df7a221086e9206652d9048fef220bc1d9bf8f0e82d5d08def1e57e358042c7f"} Sep 07 16:33:35.115786 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:35.115735 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a7d399/test-model-a7d399-kserve-56f8498599-vwp4s" event={"ID":"65552f5b-548e-4df7-a581-eae11be6962e","Type":"ContainerStarted","Data":"ddd7aca7c13c165e57ba99902c30f99590708706c624323a0063421f1268938f"} Sep 07 16:33:35.132530 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:35.132474 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-a7d399/test-model-a7d399-kserve-56f8498599-vwp4s" podStartSLOduration=1.132456153 podStartE2EDuration="1.132456153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:33:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:33:35.130865415 +0000 UTC m=+844.166088486" watchObservedRunningTime="2026-09-07 16:33:35.132456153 +0000 UTC m=+844.167679225" Sep 07 16:33:44.547327 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:44.547269 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-a7d399/test-model-a7d399-kserve-56f8498599-vwp4s" Sep 07 16:33:44.547916 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:44.547352 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-a7d399/test-model-a7d399-kserve-56f8498599-vwp4s" Sep 07 16:33:44.560301 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:44.560279 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-a7d399/test-model-a7d399-kserve-56f8498599-vwp4s" Sep 07 16:33:45.176618 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:33:45.176589 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-a7d399/test-model-a7d399-kserve-56f8498599-vwp4s" Sep 07 16:34:17.277245 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:17.277210 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cc0983-vbqgs"] Sep 07 16:34:17.277966 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:17.277660 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e124b934-cb8e-485f-8bec-4348c1307769" containerName="istio-proxy" Sep 07 16:34:17.277966 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:17.277676 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e124b934-cb8e-485f-8bec-4348c1307769" containerName="istio-proxy" Sep 07 16:34:17.280826 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:17.280811 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cc0983-vbqgs" Sep 07 16:34:17.282787 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:17.282763 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 07 16:34:17.282904 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:17.282843 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jpjxt\"" Sep 07 16:34:17.286770 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:17.286744 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cc0983-vbqgs"] Sep 07 16:34:17.394905 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:17.394861 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59916aed-5c93-4b65-b970-06c40025f94c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-cc0983-vbqgs\" (UID: \"59916aed-5c93-4b65-b970-06c40025f94c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cc0983-vbqgs" Sep 07 16:34:17.496352 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:17.496306 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59916aed-5c93-4b65-b970-06c40025f94c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-cc0983-vbqgs\" (UID: \"59916aed-5c93-4b65-b970-06c40025f94c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cc0983-vbqgs" Sep 07 16:34:17.496888 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:17.496869 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59916aed-5c93-4b65-b970-06c40025f94c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-cc0983-vbqgs\" (UID: \"59916aed-5c93-4b65-b970-06c40025f94c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cc0983-vbqgs" Sep 07 16:34:17.615916 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:17.615823 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cc0983-vbqgs" Sep 07 16:34:17.891671 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:17.891491 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cc0983-vbqgs"] Sep 07 16:34:17.894459 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:34:17.894429 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59916aed_5c93_4b65_b970_06c40025f94c.slice/crio-2180e6dee0759c7023a001bd71c676145b907459aeca48a9a0c1817afc864f2f WatchSource:0}: Error finding container 2180e6dee0759c7023a001bd71c676145b907459aeca48a9a0c1817afc864f2f: Status 404 returned error can't find the container with id 2180e6dee0759c7023a001bd71c676145b907459aeca48a9a0c1817afc864f2f Sep 07 16:34:18.322075 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:18.322042 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cc0983-vbqgs" event={"ID":"59916aed-5c93-4b65-b970-06c40025f94c","Type":"ContainerStarted","Data":"188d01da977d39ac9a72bd18aaa104e38e5e18adae3256788b5d8e4adce510e3"} Sep 07 16:34:18.322530 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:18.322090 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cc0983-vbqgs" event={"ID":"59916aed-5c93-4b65-b970-06c40025f94c","Type":"ContainerStarted","Data":"2180e6dee0759c7023a001bd71c676145b907459aeca48a9a0c1817afc864f2f"} Sep 07 16:34:18.335534 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:18.335480 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cc0983-vbqgs" podStartSLOduration=1.3354629199999999 podStartE2EDuration="1.33546292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:34:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:34:18.33456514 +0000 UTC m=+887.369788204" watchObservedRunningTime="2026-09-07 16:34:18.33546292 +0000 UTC m=+887.370685991" Sep 07 16:34:19.328712 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:19.328674 2580 generic.go:356] "Generic (PLEG): container finished" podID="59916aed-5c93-4b65-b970-06c40025f94c" containerID="188d01da977d39ac9a72bd18aaa104e38e5e18adae3256788b5d8e4adce510e3" exitCode=0 Sep 07 16:34:19.329190 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:19.328744 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cc0983-vbqgs" event={"ID":"59916aed-5c93-4b65-b970-06c40025f94c","Type":"ContainerDied","Data":"188d01da977d39ac9a72bd18aaa104e38e5e18adae3256788b5d8e4adce510e3"} Sep 07 16:34:20.467057 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:20.467012 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cc0983-vbqgs" Sep 07 16:34:20.522381 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:20.522349 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59916aed-5c93-4b65-b970-06c40025f94c-maas-api-service-ca\") pod \"59916aed-5c93-4b65-b970-06c40025f94c\" (UID: \"59916aed-5c93-4b65-b970-06c40025f94c\") " Sep 07 16:34:20.522716 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:20.522682 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59916aed-5c93-4b65-b970-06c40025f94c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "59916aed-5c93-4b65-b970-06c40025f94c" (UID: "59916aed-5c93-4b65-b970-06c40025f94c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 16:34:20.623057 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:20.622958 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59916aed-5c93-4b65-b970-06c40025f94c-maas-api-service-ca\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:34:21.339869 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:21.339832 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cc0983-vbqgs" event={"ID":"59916aed-5c93-4b65-b970-06c40025f94c","Type":"ContainerDied","Data":"2180e6dee0759c7023a001bd71c676145b907459aeca48a9a0c1817afc864f2f"} Sep 07 16:34:21.340043 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:21.339879 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2180e6dee0759c7023a001bd71c676145b907459aeca48a9a0c1817afc864f2f" Sep 07 16:34:21.340043 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:21.339847 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cc0983-vbqgs" Sep 07 16:34:27.905878 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:27.905746 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-cc0983-7978788689-9p9gm"] Sep 07 16:34:27.906618 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:27.906212 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-cc0983-7978788689-9p9gm" podUID="084f00fe-6d3f-460c-846d-8a9751ec216d" containerName="payload-processing" containerID="cri-o://aa16ecd53221719f009e8ac60edddfcef280878552a68faebdb32b157b23b90a" gracePeriod=30 Sep 07 16:34:28.067523 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:28.067485 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cc0983-vbqgs"] Sep 07 16:34:28.070057 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:28.070016 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cc0983-vbqgs"] Sep 07 16:34:28.154376 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:28.154350 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-cc0983-7978788689-9p9gm" Sep 07 16:34:28.291648 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:28.291615 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v5972\" (UniqueName: \"kubernetes.io/projected/084f00fe-6d3f-460c-846d-8a9751ec216d-kube-api-access-v5972\") pod \"084f00fe-6d3f-460c-846d-8a9751ec216d\" (UID: \"084f00fe-6d3f-460c-846d-8a9751ec216d\") " Sep 07 16:34:28.291862 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:28.291672 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/084f00fe-6d3f-460c-846d-8a9751ec216d-plugins-config-volume\") pod \"084f00fe-6d3f-460c-846d-8a9751ec216d\" (UID: \"084f00fe-6d3f-460c-846d-8a9751ec216d\") " Sep 07 16:34:28.292111 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:28.292082 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/084f00fe-6d3f-460c-846d-8a9751ec216d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "084f00fe-6d3f-460c-846d-8a9751ec216d" (UID: "084f00fe-6d3f-460c-846d-8a9751ec216d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 16:34:28.293970 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:28.293939 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/084f00fe-6d3f-460c-846d-8a9751ec216d-kube-api-access-v5972" (OuterVolumeSpecName: "kube-api-access-v5972") pod "084f00fe-6d3f-460c-846d-8a9751ec216d" (UID: "084f00fe-6d3f-460c-846d-8a9751ec216d"). InnerVolumeSpecName "kube-api-access-v5972". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 16:34:28.373975 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:28.373937 2580 generic.go:356] "Generic (PLEG): container finished" podID="084f00fe-6d3f-460c-846d-8a9751ec216d" containerID="aa16ecd53221719f009e8ac60edddfcef280878552a68faebdb32b157b23b90a" exitCode=0 Sep 07 16:34:28.374169 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:28.373983 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-cc0983-7978788689-9p9gm" Sep 07 16:34:28.374169 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:28.374055 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-cc0983-7978788689-9p9gm" event={"ID":"084f00fe-6d3f-460c-846d-8a9751ec216d","Type":"ContainerDied","Data":"aa16ecd53221719f009e8ac60edddfcef280878552a68faebdb32b157b23b90a"} Sep 07 16:34:28.374169 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:28.374107 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-cc0983-7978788689-9p9gm" event={"ID":"084f00fe-6d3f-460c-846d-8a9751ec216d","Type":"ContainerDied","Data":"1c081532dfe9c124e1a6a3466cd77a9ec078068660f97aee93894220b4cd33cc"} Sep 07 16:34:28.374169 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:28.374130 2580 scope.go:117] "RemoveContainer" containerID="aa16ecd53221719f009e8ac60edddfcef280878552a68faebdb32b157b23b90a" Sep 07 16:34:28.382948 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:28.382932 2580 scope.go:117] "RemoveContainer" containerID="aa16ecd53221719f009e8ac60edddfcef280878552a68faebdb32b157b23b90a" Sep 07 16:34:28.383250 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:34:28.383231 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa16ecd53221719f009e8ac60edddfcef280878552a68faebdb32b157b23b90a\": container with ID starting with aa16ecd53221719f009e8ac60edddfcef280878552a68faebdb32b157b23b90a not found: ID does not exist" containerID="aa16ecd53221719f009e8ac60edddfcef280878552a68faebdb32b157b23b90a" Sep 07 16:34:28.383308 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:28.383259 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa16ecd53221719f009e8ac60edddfcef280878552a68faebdb32b157b23b90a"} err="failed to get container status \"aa16ecd53221719f009e8ac60edddfcef280878552a68faebdb32b157b23b90a\": rpc error: code = NotFound desc = could not find container \"aa16ecd53221719f009e8ac60edddfcef280878552a68faebdb32b157b23b90a\": container with ID starting with aa16ecd53221719f009e8ac60edddfcef280878552a68faebdb32b157b23b90a not found: ID does not exist" Sep 07 16:34:28.392641 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:28.392618 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5972\" (UniqueName: \"kubernetes.io/projected/084f00fe-6d3f-460c-846d-8a9751ec216d-kube-api-access-v5972\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:34:28.392641 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:28.392639 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/084f00fe-6d3f-460c-846d-8a9751ec216d-plugins-config-volume\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:34:28.394099 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:28.394076 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-cc0983-7978788689-9p9gm"] Sep 07 16:34:28.398321 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:28.398294 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-cc0983-7978788689-9p9gm"] Sep 07 16:34:29.417906 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:29.417862 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="084f00fe-6d3f-460c-846d-8a9751ec216d" path="/var/lib/kubelet/pods/084f00fe-6d3f-460c-846d-8a9751ec216d/volumes" Sep 07 16:34:29.424051 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:29.423985 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59916aed-5c93-4b65-b970-06c40025f94c" path="/var/lib/kubelet/pods/59916aed-5c93-4b65-b970-06c40025f94c/volumes" Sep 07 16:34:35.046989 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:35.046947 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw"] Sep 07 16:34:35.047563 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:35.047337 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59916aed-5c93-4b65-b970-06c40025f94c" containerName="revoke-keys" Sep 07 16:34:35.047563 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:35.047352 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="59916aed-5c93-4b65-b970-06c40025f94c" containerName="revoke-keys" Sep 07 16:34:35.047563 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:35.047373 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="084f00fe-6d3f-460c-846d-8a9751ec216d" containerName="payload-processing" Sep 07 16:34:35.047563 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:35.047379 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="084f00fe-6d3f-460c-846d-8a9751ec216d" containerName="payload-processing" Sep 07 16:34:35.047563 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:35.047443 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="084f00fe-6d3f-460c-846d-8a9751ec216d" containerName="payload-processing" Sep 07 16:34:35.047563 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:35.047451 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="59916aed-5c93-4b65-b970-06c40025f94c" containerName="revoke-keys" Sep 07 16:34:35.052299 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:35.052274 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw" Sep 07 16:34:35.054353 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:35.054333 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-8a18f8-openshift-default-dockercfg-88b4b\"" Sep 07 16:34:35.063334 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:35.063308 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw"] Sep 07 16:34:35.157503 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:35.157466 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4aac6863-1edf-444d-a9d9-673bc3e80b82-istio-podinfo\") pod \"e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw\" (UID: \"4aac6863-1edf-444d-a9d9-673bc3e80b82\") " pod="openshift-ingress/e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw" Sep 07 16:34:35.157503 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:35.157511 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aac6863-1edf-444d-a9d9-673bc3e80b82-credential-socket\") pod \"e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw\" (UID: \"4aac6863-1edf-444d-a9d9-673bc3e80b82\") " pod="openshift-ingress/e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw" Sep 07 16:34:35.157737 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:35.157533 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aac6863-1edf-444d-a9d9-673bc3e80b82-workload-socket\") pod \"e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw\" (UID: \"4aac6863-1edf-444d-a9d9-673bc3e80b82\") " pod="openshift-ingress/e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw" Sep 07 16:34:35.157737 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:35.157608 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4aac6863-1edf-444d-a9d9-673bc3e80b82-istio-data\") pod \"e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw\" (UID: \"4aac6863-1edf-444d-a9d9-673bc3e80b82\") " pod="openshift-ingress/e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw" Sep 07 16:34:35.157737 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:35.157682 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4aac6863-1edf-444d-a9d9-673bc3e80b82-istiod-ca-cert\") pod \"e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw\" (UID: \"4aac6863-1edf-444d-a9d9-673bc3e80b82\") " pod="openshift-ingress/e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw" Sep 07 16:34:35.157737 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:35.157725 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4aac6863-1edf-444d-a9d9-673bc3e80b82-istio-envoy\") pod \"e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw\" (UID: \"4aac6863-1edf-444d-a9d9-673bc3e80b82\") " pod="openshift-ingress/e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw" Sep 07 16:34:35.157884 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:35.157740 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4aac6863-1edf-444d-a9d9-673bc3e80b82-istio-token\") pod \"e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw\" (UID: \"4aac6863-1edf-444d-a9d9-673bc3e80b82\") " pod="openshift-ingress/e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw" Sep 07 16:34:35.157884 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:35.157768 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv57s\" (UniqueName: \"kubernetes.io/projected/4aac6863-1edf-444d-a9d9-673bc3e80b82-kube-api-access-tv57s\") pod \"e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw\" (UID: \"4aac6863-1edf-444d-a9d9-673bc3e80b82\") " pod="openshift-ingress/e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw" Sep 07 16:34:35.157884 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:35.157819 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4aac6863-1edf-444d-a9d9-673bc3e80b82-workload-certs\") pod \"e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw\" (UID: \"4aac6863-1edf-444d-a9d9-673bc3e80b82\") " pod="openshift-ingress/e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw" Sep 07 16:34:35.258319 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:35.258273 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4aac6863-1edf-444d-a9d9-673bc3e80b82-istio-data\") pod \"e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw\" (UID: \"4aac6863-1edf-444d-a9d9-673bc3e80b82\") " pod="openshift-ingress/e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw" Sep 07 16:34:35.258522 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:35.258356 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4aac6863-1edf-444d-a9d9-673bc3e80b82-istiod-ca-cert\") pod \"e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw\" (UID: \"4aac6863-1edf-444d-a9d9-673bc3e80b82\") " pod="openshift-ingress/e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw" Sep 07 16:34:35.258522 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:35.258394 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4aac6863-1edf-444d-a9d9-673bc3e80b82-istio-envoy\") pod \"e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw\" (UID: \"4aac6863-1edf-444d-a9d9-673bc3e80b82\") " pod="openshift-ingress/e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw" Sep 07 16:34:35.258522 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:35.258409 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4aac6863-1edf-444d-a9d9-673bc3e80b82-istio-token\") pod \"e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw\" (UID: \"4aac6863-1edf-444d-a9d9-673bc3e80b82\") " pod="openshift-ingress/e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw" Sep 07 16:34:35.258522 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:35.258437 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv57s\" (UniqueName: \"kubernetes.io/projected/4aac6863-1edf-444d-a9d9-673bc3e80b82-kube-api-access-tv57s\") pod \"e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw\" (UID: \"4aac6863-1edf-444d-a9d9-673bc3e80b82\") " pod="openshift-ingress/e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw" Sep 07 16:34:35.258522 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:35.258464 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4aac6863-1edf-444d-a9d9-673bc3e80b82-workload-certs\") pod \"e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw\" (UID: \"4aac6863-1edf-444d-a9d9-673bc3e80b82\") " pod="openshift-ingress/e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw" Sep 07 16:34:35.258522 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:35.258507 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4aac6863-1edf-444d-a9d9-673bc3e80b82-istio-podinfo\") pod \"e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw\" (UID: \"4aac6863-1edf-444d-a9d9-673bc3e80b82\") " pod="openshift-ingress/e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw" Sep 07 16:34:35.258868 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:35.258668 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aac6863-1edf-444d-a9d9-673bc3e80b82-credential-socket\") pod \"e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw\" (UID: \"4aac6863-1edf-444d-a9d9-673bc3e80b82\") " pod="openshift-ingress/e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw" Sep 07 16:34:35.258868 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:35.258704 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aac6863-1edf-444d-a9d9-673bc3e80b82-workload-socket\") pod \"e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw\" (UID: \"4aac6863-1edf-444d-a9d9-673bc3e80b82\") " pod="openshift-ingress/e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw" Sep 07 16:34:35.258868 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:35.258746 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4aac6863-1edf-444d-a9d9-673bc3e80b82-istio-data\") pod \"e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw\" (UID: \"4aac6863-1edf-444d-a9d9-673bc3e80b82\") " pod="openshift-ingress/e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw" Sep 07 16:34:35.259064 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:35.258999 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aac6863-1edf-444d-a9d9-673bc3e80b82-credential-socket\") pod \"e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw\" (UID: \"4aac6863-1edf-444d-a9d9-673bc3e80b82\") " pod="openshift-ingress/e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw" Sep 07 16:34:35.259137 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:35.259099 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aac6863-1edf-444d-a9d9-673bc3e80b82-workload-socket\") pod \"e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw\" (UID: \"4aac6863-1edf-444d-a9d9-673bc3e80b82\") " pod="openshift-ingress/e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw" Sep 07 16:34:35.259137 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:35.259128 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4aac6863-1edf-444d-a9d9-673bc3e80b82-istiod-ca-cert\") pod \"e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw\" (UID: \"4aac6863-1edf-444d-a9d9-673bc3e80b82\") " pod="openshift-ingress/e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw" Sep 07 16:34:35.259248 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:35.259159 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4aac6863-1edf-444d-a9d9-673bc3e80b82-workload-certs\") pod \"e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw\" (UID: \"4aac6863-1edf-444d-a9d9-673bc3e80b82\") " pod="openshift-ingress/e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw" Sep 07 16:34:35.261255 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:35.261232 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4aac6863-1edf-444d-a9d9-673bc3e80b82-istio-envoy\") pod \"e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw\" (UID: \"4aac6863-1edf-444d-a9d9-673bc3e80b82\") " pod="openshift-ingress/e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw" Sep 07 16:34:35.261364 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:35.261286 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4aac6863-1edf-444d-a9d9-673bc3e80b82-istio-podinfo\") pod \"e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw\" (UID: \"4aac6863-1edf-444d-a9d9-673bc3e80b82\") " pod="openshift-ingress/e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw" Sep 07 16:34:35.265921 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:35.265895 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv57s\" (UniqueName: \"kubernetes.io/projected/4aac6863-1edf-444d-a9d9-673bc3e80b82-kube-api-access-tv57s\") pod \"e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw\" (UID: \"4aac6863-1edf-444d-a9d9-673bc3e80b82\") " pod="openshift-ingress/e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw" Sep 07 16:34:35.266013 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:35.265962 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4aac6863-1edf-444d-a9d9-673bc3e80b82-istio-token\") pod \"e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw\" (UID: \"4aac6863-1edf-444d-a9d9-673bc3e80b82\") " pod="openshift-ingress/e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw" Sep 07 16:34:35.363867 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:35.363773 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw" Sep 07 16:34:35.641906 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:35.641821 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw"] Sep 07 16:34:35.644913 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:34:35.644881 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aac6863_1edf_444d_a9d9_673bc3e80b82.slice/crio-d73a9fa8fedbd474ffa943a540861cb0c0ed077cc4212ad97138d3a0c0294a45 WatchSource:0}: Error finding container d73a9fa8fedbd474ffa943a540861cb0c0ed077cc4212ad97138d3a0c0294a45: Status 404 returned error can't find the container with id d73a9fa8fedbd474ffa943a540861cb0c0ed077cc4212ad97138d3a0c0294a45 Sep 07 16:34:35.647209 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:35.647175 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:34:35.647306 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:35.647245 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:34:35.647306 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:35.647276 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:34:36.414646 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:36.414607 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw" event={"ID":"4aac6863-1edf-444d-a9d9-673bc3e80b82","Type":"ContainerStarted","Data":"575e5bfb10a5e8002a08a1f4d2ba7c0ee897cdf6174ea0babba278148a050919"} Sep 07 16:34:36.415180 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:36.414659 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw" event={"ID":"4aac6863-1edf-444d-a9d9-673bc3e80b82","Type":"ContainerStarted","Data":"d73a9fa8fedbd474ffa943a540861cb0c0ed077cc4212ad97138d3a0c0294a45"} Sep 07 16:34:36.433985 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:36.433910 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw" podStartSLOduration=1.433891928 podStartE2EDuration="1.433891928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:34:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:34:36.431664845 +0000 UTC m=+905.466887916" watchObservedRunningTime="2026-09-07 16:34:36.433891928 +0000 UTC m=+905.469114999" Sep 07 16:34:37.364711 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:37.364668 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw" Sep 07 16:34:37.369865 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:37.369838 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw" Sep 07 16:34:37.423674 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:37.423641 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw" Sep 07 16:34:37.424582 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:37.424561 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw" Sep 07 16:34:57.556805 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:57.556772 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw"] Sep 07 16:34:57.557373 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:57.557097 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw" podUID="4aac6863-1edf-444d-a9d9-673bc3e80b82" containerName="istio-proxy" containerID="cri-o://575e5bfb10a5e8002a08a1f4d2ba7c0ee897cdf6174ea0babba278148a050919" gracePeriod=30 Sep 07 16:34:58.267008 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:58.266967 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm"] Sep 07 16:34:58.270839 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:58.270816 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm" Sep 07 16:34:58.272985 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:58.272960 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-65f07c0f-openshift-default-dockercfg-c2jgb\"" Sep 07 16:34:58.284911 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:58.284887 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm"] Sep 07 16:34:58.360517 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:58.360469 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xflj4\" (UniqueName: \"kubernetes.io/projected/275283fc-2837-4fe0-884b-95803aac56c4-kube-api-access-xflj4\") pod \"e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm\" (UID: \"275283fc-2837-4fe0-884b-95803aac56c4\") " pod="openshift-ingress/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm" Sep 07 16:34:58.360740 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:58.360594 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/275283fc-2837-4fe0-884b-95803aac56c4-workload-certs\") pod \"e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm\" (UID: \"275283fc-2837-4fe0-884b-95803aac56c4\") " pod="openshift-ingress/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm" Sep 07 16:34:58.360740 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:58.360633 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/275283fc-2837-4fe0-884b-95803aac56c4-credential-socket\") pod \"e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm\" (UID: \"275283fc-2837-4fe0-884b-95803aac56c4\") " pod="openshift-ingress/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm" Sep 07 16:34:58.360740 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:58.360670 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/275283fc-2837-4fe0-884b-95803aac56c4-istio-envoy\") pod \"e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm\" (UID: \"275283fc-2837-4fe0-884b-95803aac56c4\") " pod="openshift-ingress/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm" Sep 07 16:34:58.360740 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:58.360696 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/275283fc-2837-4fe0-884b-95803aac56c4-workload-socket\") pod \"e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm\" (UID: \"275283fc-2837-4fe0-884b-95803aac56c4\") " pod="openshift-ingress/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm" Sep 07 16:34:58.360966 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:58.360772 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/275283fc-2837-4fe0-884b-95803aac56c4-istiod-ca-cert\") pod \"e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm\" (UID: \"275283fc-2837-4fe0-884b-95803aac56c4\") " pod="openshift-ingress/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm" Sep 07 16:34:58.360966 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:58.360822 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/275283fc-2837-4fe0-884b-95803aac56c4-istio-data\") pod \"e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm\" (UID: \"275283fc-2837-4fe0-884b-95803aac56c4\") " pod="openshift-ingress/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm" Sep 07 16:34:58.360966 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:58.360863 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/275283fc-2837-4fe0-884b-95803aac56c4-istio-token\") pod \"e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm\" (UID: \"275283fc-2837-4fe0-884b-95803aac56c4\") " pod="openshift-ingress/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm" Sep 07 16:34:58.360966 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:58.360892 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/275283fc-2837-4fe0-884b-95803aac56c4-istio-podinfo\") pod \"e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm\" (UID: \"275283fc-2837-4fe0-884b-95803aac56c4\") " pod="openshift-ingress/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm" Sep 07 16:34:58.462317 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:58.462263 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/275283fc-2837-4fe0-884b-95803aac56c4-workload-socket\") pod \"e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm\" (UID: \"275283fc-2837-4fe0-884b-95803aac56c4\") " pod="openshift-ingress/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm" Sep 07 16:34:58.462317 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:58.462321 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/275283fc-2837-4fe0-884b-95803aac56c4-istiod-ca-cert\") pod \"e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm\" (UID: \"275283fc-2837-4fe0-884b-95803aac56c4\") " pod="openshift-ingress/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm" Sep 07 16:34:58.462569 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:58.462346 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/275283fc-2837-4fe0-884b-95803aac56c4-istio-data\") pod \"e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm\" (UID: \"275283fc-2837-4fe0-884b-95803aac56c4\") " pod="openshift-ingress/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm" Sep 07 16:34:58.462569 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:58.462373 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/275283fc-2837-4fe0-884b-95803aac56c4-istio-token\") pod \"e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm\" (UID: \"275283fc-2837-4fe0-884b-95803aac56c4\") " pod="openshift-ingress/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm" Sep 07 16:34:58.462569 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:58.462394 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/275283fc-2837-4fe0-884b-95803aac56c4-istio-podinfo\") pod \"e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm\" (UID: \"275283fc-2837-4fe0-884b-95803aac56c4\") " pod="openshift-ingress/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm" Sep 07 16:34:58.462569 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:58.462435 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xflj4\" (UniqueName: \"kubernetes.io/projected/275283fc-2837-4fe0-884b-95803aac56c4-kube-api-access-xflj4\") pod \"e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm\" (UID: \"275283fc-2837-4fe0-884b-95803aac56c4\") " pod="openshift-ingress/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm" Sep 07 16:34:58.462569 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:58.462528 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/275283fc-2837-4fe0-884b-95803aac56c4-workload-certs\") pod \"e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm\" (UID: \"275283fc-2837-4fe0-884b-95803aac56c4\") " pod="openshift-ingress/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm" Sep 07 16:34:58.462860 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:58.462578 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/275283fc-2837-4fe0-884b-95803aac56c4-credential-socket\") pod \"e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm\" (UID: \"275283fc-2837-4fe0-884b-95803aac56c4\") " pod="openshift-ingress/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm" Sep 07 16:34:58.462860 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:58.462630 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/275283fc-2837-4fe0-884b-95803aac56c4-istio-envoy\") pod \"e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm\" (UID: \"275283fc-2837-4fe0-884b-95803aac56c4\") " pod="openshift-ingress/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm" Sep 07 16:34:58.462860 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:58.462732 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/275283fc-2837-4fe0-884b-95803aac56c4-workload-socket\") pod \"e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm\" (UID: \"275283fc-2837-4fe0-884b-95803aac56c4\") " pod="openshift-ingress/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm" Sep 07 16:34:58.462860 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:58.462810 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/275283fc-2837-4fe0-884b-95803aac56c4-istio-data\") pod \"e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm\" (UID: \"275283fc-2837-4fe0-884b-95803aac56c4\") " pod="openshift-ingress/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm" Sep 07 16:34:58.463112 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:58.463081 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/275283fc-2837-4fe0-884b-95803aac56c4-credential-socket\") pod \"e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm\" (UID: \"275283fc-2837-4fe0-884b-95803aac56c4\") " pod="openshift-ingress/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm" Sep 07 16:34:58.463182 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:58.463140 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/275283fc-2837-4fe0-884b-95803aac56c4-istiod-ca-cert\") pod \"e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm\" (UID: \"275283fc-2837-4fe0-884b-95803aac56c4\") " pod="openshift-ingress/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm" Sep 07 16:34:58.463245 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:58.463218 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/275283fc-2837-4fe0-884b-95803aac56c4-workload-certs\") pod \"e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm\" (UID: \"275283fc-2837-4fe0-884b-95803aac56c4\") " pod="openshift-ingress/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm" Sep 07 16:34:58.465182 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:58.465153 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/275283fc-2837-4fe0-884b-95803aac56c4-istio-envoy\") pod \"e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm\" (UID: \"275283fc-2837-4fe0-884b-95803aac56c4\") " pod="openshift-ingress/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm" Sep 07 16:34:58.465605 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:58.465585 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/275283fc-2837-4fe0-884b-95803aac56c4-istio-podinfo\") pod \"e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm\" (UID: \"275283fc-2837-4fe0-884b-95803aac56c4\") " pod="openshift-ingress/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm" Sep 07 16:34:58.470281 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:58.470256 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/275283fc-2837-4fe0-884b-95803aac56c4-istio-token\") pod \"e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm\" (UID: \"275283fc-2837-4fe0-884b-95803aac56c4\") " pod="openshift-ingress/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm" Sep 07 16:34:58.470782 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:58.470752 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xflj4\" (UniqueName: \"kubernetes.io/projected/275283fc-2837-4fe0-884b-95803aac56c4-kube-api-access-xflj4\") pod \"e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm\" (UID: \"275283fc-2837-4fe0-884b-95803aac56c4\") " pod="openshift-ingress/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm" Sep 07 16:34:58.584933 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:58.584826 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm" Sep 07 16:34:58.864340 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:58.864317 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm"] Sep 07 16:34:58.866689 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:34:58.866660 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod275283fc_2837_4fe0_884b_95803aac56c4.slice/crio-fe6bb6ccb4482df530cdb342bba0cd06ad1e42a005b201ce1181228142c0f80f WatchSource:0}: Error finding container fe6bb6ccb4482df530cdb342bba0cd06ad1e42a005b201ce1181228142c0f80f: Status 404 returned error can't find the container with id fe6bb6ccb4482df530cdb342bba0cd06ad1e42a005b201ce1181228142c0f80f Sep 07 16:34:58.869127 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:58.869087 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:34:58.869221 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:58.869180 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:34:58.869261 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:58.869227 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:34:59.528798 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:59.528763 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm" event={"ID":"275283fc-2837-4fe0-884b-95803aac56c4","Type":"ContainerStarted","Data":"a60850c31e11e979bdb5c12235b8d4a4828ecc94fe465ad39ec71505615cc201"} Sep 07 16:34:59.528983 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:59.528829 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm" event={"ID":"275283fc-2837-4fe0-884b-95803aac56c4","Type":"ContainerStarted","Data":"fe6bb6ccb4482df530cdb342bba0cd06ad1e42a005b201ce1181228142c0f80f"} Sep 07 16:34:59.548898 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:59.548843 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm" podStartSLOduration=1.548829103 podStartE2EDuration="1.548829103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:34:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:34:59.547228371 +0000 UTC m=+928.582451440" watchObservedRunningTime="2026-09-07 16:34:59.548829103 +0000 UTC m=+928.584052174" Sep 07 16:34:59.585994 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:59.585960 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm" Sep 07 16:34:59.587809 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:59.587778 2580 patch_prober.go:28] interesting pod/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Sep 07 16:34:59.587939 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:34:59.587840 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm" podUID="275283fc-2837-4fe0-884b-95803aac56c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Sep 07 16:35:00.589764 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:00.589732 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm" Sep 07 16:35:01.538627 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:01.538586 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm" Sep 07 16:35:01.539718 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:01.539700 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm" Sep 07 16:35:02.814431 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:02.814402 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw" Sep 07 16:35:02.903340 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:02.903244 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4aac6863-1edf-444d-a9d9-673bc3e80b82-istio-data\") pod \"4aac6863-1edf-444d-a9d9-673bc3e80b82\" (UID: \"4aac6863-1edf-444d-a9d9-673bc3e80b82\") " Sep 07 16:35:02.903340 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:02.903325 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4aac6863-1edf-444d-a9d9-673bc3e80b82-istiod-ca-cert\") pod \"4aac6863-1edf-444d-a9d9-673bc3e80b82\" (UID: \"4aac6863-1edf-444d-a9d9-673bc3e80b82\") " Sep 07 16:35:02.903689 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:02.903355 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tv57s\" (UniqueName: \"kubernetes.io/projected/4aac6863-1edf-444d-a9d9-673bc3e80b82-kube-api-access-tv57s\") pod \"4aac6863-1edf-444d-a9d9-673bc3e80b82\" (UID: \"4aac6863-1edf-444d-a9d9-673bc3e80b82\") " Sep 07 16:35:02.903689 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:02.903379 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aac6863-1edf-444d-a9d9-673bc3e80b82-workload-socket\") pod \"4aac6863-1edf-444d-a9d9-673bc3e80b82\" (UID: \"4aac6863-1edf-444d-a9d9-673bc3e80b82\") " Sep 07 16:35:02.903689 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:02.903402 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4aac6863-1edf-444d-a9d9-673bc3e80b82-istio-envoy\") pod \"4aac6863-1edf-444d-a9d9-673bc3e80b82\" (UID: \"4aac6863-1edf-444d-a9d9-673bc3e80b82\") " Sep 07 16:35:02.903689 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:02.903435 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aac6863-1edf-444d-a9d9-673bc3e80b82-credential-socket\") pod \"4aac6863-1edf-444d-a9d9-673bc3e80b82\" (UID: \"4aac6863-1edf-444d-a9d9-673bc3e80b82\") " Sep 07 16:35:02.903689 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:02.903456 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4aac6863-1edf-444d-a9d9-673bc3e80b82-istio-token\") pod \"4aac6863-1edf-444d-a9d9-673bc3e80b82\" (UID: \"4aac6863-1edf-444d-a9d9-673bc3e80b82\") " Sep 07 16:35:02.903689 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:02.903535 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4aac6863-1edf-444d-a9d9-673bc3e80b82-istio-podinfo\") pod \"4aac6863-1edf-444d-a9d9-673bc3e80b82\" (UID: \"4aac6863-1edf-444d-a9d9-673bc3e80b82\") " Sep 07 16:35:02.903689 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:02.903557 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4aac6863-1edf-444d-a9d9-673bc3e80b82-workload-certs\") pod \"4aac6863-1edf-444d-a9d9-673bc3e80b82\" (UID: \"4aac6863-1edf-444d-a9d9-673bc3e80b82\") " Sep 07 16:35:02.903689 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:02.903567 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aac6863-1edf-444d-a9d9-673bc3e80b82-istio-data" (OuterVolumeSpecName: "istio-data") pod "4aac6863-1edf-444d-a9d9-673bc3e80b82" (UID: "4aac6863-1edf-444d-a9d9-673bc3e80b82"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 16:35:02.904195 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:02.903703 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aac6863-1edf-444d-a9d9-673bc3e80b82-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4aac6863-1edf-444d-a9d9-673bc3e80b82" (UID: "4aac6863-1edf-444d-a9d9-673bc3e80b82"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 16:35:02.904195 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:02.903746 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aac6863-1edf-444d-a9d9-673bc3e80b82-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4aac6863-1edf-444d-a9d9-673bc3e80b82" (UID: "4aac6863-1edf-444d-a9d9-673bc3e80b82"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 16:35:02.904195 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:02.903912 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aac6863-1edf-444d-a9d9-673bc3e80b82-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4aac6863-1edf-444d-a9d9-673bc3e80b82" (UID: "4aac6863-1edf-444d-a9d9-673bc3e80b82"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 16:35:02.904195 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:02.903944 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aac6863-1edf-444d-a9d9-673bc3e80b82-credential-socket\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:35:02.904195 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:02.903967 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4aac6863-1edf-444d-a9d9-673bc3e80b82-istio-data\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:35:02.904195 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:02.903982 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4aac6863-1edf-444d-a9d9-673bc3e80b82-istiod-ca-cert\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:35:02.904195 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:02.903991 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aac6863-1edf-444d-a9d9-673bc3e80b82-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4aac6863-1edf-444d-a9d9-673bc3e80b82" (UID: "4aac6863-1edf-444d-a9d9-673bc3e80b82"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 16:35:02.905869 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:02.905842 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aac6863-1edf-444d-a9d9-673bc3e80b82-istio-token" (OuterVolumeSpecName: "istio-token") pod "4aac6863-1edf-444d-a9d9-673bc3e80b82" (UID: "4aac6863-1edf-444d-a9d9-673bc3e80b82"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 16:35:02.906036 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:02.905864 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aac6863-1edf-444d-a9d9-673bc3e80b82-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4aac6863-1edf-444d-a9d9-673bc3e80b82" (UID: "4aac6863-1edf-444d-a9d9-673bc3e80b82"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 16:35:02.906036 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:02.905913 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4aac6863-1edf-444d-a9d9-673bc3e80b82-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4aac6863-1edf-444d-a9d9-673bc3e80b82" (UID: "4aac6863-1edf-444d-a9d9-673bc3e80b82"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 07 16:35:02.906036 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:02.905971 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aac6863-1edf-444d-a9d9-673bc3e80b82-kube-api-access-tv57s" (OuterVolumeSpecName: "kube-api-access-tv57s") pod "4aac6863-1edf-444d-a9d9-673bc3e80b82" (UID: "4aac6863-1edf-444d-a9d9-673bc3e80b82"). InnerVolumeSpecName "kube-api-access-tv57s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 16:35:03.004796 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:03.004751 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aac6863-1edf-444d-a9d9-673bc3e80b82-workload-socket\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:35:03.004796 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:03.004789 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4aac6863-1edf-444d-a9d9-673bc3e80b82-istio-envoy\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:35:03.004796 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:03.004801 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4aac6863-1edf-444d-a9d9-673bc3e80b82-istio-token\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:35:03.005089 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:03.004812 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4aac6863-1edf-444d-a9d9-673bc3e80b82-istio-podinfo\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:35:03.005089 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:03.004824 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4aac6863-1edf-444d-a9d9-673bc3e80b82-workload-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:35:03.005089 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:03.004835 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tv57s\" (UniqueName: \"kubernetes.io/projected/4aac6863-1edf-444d-a9d9-673bc3e80b82-kube-api-access-tv57s\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:35:03.344768 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:03.344730 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-65f07c0f-6ffc48f886-gmprf"] Sep 07 16:35:03.345127 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:03.345110 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4aac6863-1edf-444d-a9d9-673bc3e80b82" containerName="istio-proxy" Sep 07 16:35:03.345177 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:03.345132 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4aac6863-1edf-444d-a9d9-673bc3e80b82" containerName="istio-proxy" Sep 07 16:35:03.345217 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:03.345188 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4aac6863-1edf-444d-a9d9-673bc3e80b82" containerName="istio-proxy" Sep 07 16:35:03.348194 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:03.348175 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-65f07c0f-6ffc48f886-gmprf" Sep 07 16:35:03.350393 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:03.350364 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-65f07c0f\"" Sep 07 16:35:03.350522 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:03.350413 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-65f07c0f-dockercfg-4v9jw\"" Sep 07 16:35:03.356648 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:03.356626 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-65f07c0f-6ffc48f886-gmprf"] Sep 07 16:35:03.363361 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:03.363321 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-65f07c0f-7f7d7bf96f-5b7l2"] Sep 07 16:35:03.366622 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:03.366601 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-65f07c0f-7f7d7bf96f-5b7l2" Sep 07 16:35:03.378223 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:03.378194 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-65f07c0f-7f7d7bf96f-5b7l2"] Sep 07 16:35:03.409102 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:03.409066 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79ac266c-6176-4cb9-8486-05f777b73bc9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-65f07c0f-6ffc48f886-gmprf\" (UID: \"79ac266c-6176-4cb9-8486-05f777b73bc9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-65f07c0f-6ffc48f886-gmprf" Sep 07 16:35:03.409284 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:03.409138 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f67j8\" (UniqueName: \"kubernetes.io/projected/6aa65db8-d977-426d-8336-aa870a95fb4c-kube-api-access-f67j8\") pod \"payload-processing-e2e-mt-65f07c0f-7f7d7bf96f-5b7l2\" (UID: \"6aa65db8-d977-426d-8336-aa870a95fb4c\") " pod="openshift-ingress/payload-processing-e2e-mt-65f07c0f-7f7d7bf96f-5b7l2" Sep 07 16:35:03.409284 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:03.409199 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6aa65db8-d977-426d-8336-aa870a95fb4c-plugins-config-volume\") pod \"payload-processing-e2e-mt-65f07c0f-7f7d7bf96f-5b7l2\" (UID: \"6aa65db8-d977-426d-8336-aa870a95fb4c\") " pod="openshift-ingress/payload-processing-e2e-mt-65f07c0f-7f7d7bf96f-5b7l2" Sep 07 16:35:03.409367 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:03.409290 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7279z\" (UniqueName: \"kubernetes.io/projected/79ac266c-6176-4cb9-8486-05f777b73bc9-kube-api-access-7279z\") pod \"payload-pre-processing-e2e-mt-65f07c0f-6ffc48f886-gmprf\" (UID: \"79ac266c-6176-4cb9-8486-05f777b73bc9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-65f07c0f-6ffc48f886-gmprf" Sep 07 16:35:03.510627 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:03.510585 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f67j8\" (UniqueName: \"kubernetes.io/projected/6aa65db8-d977-426d-8336-aa870a95fb4c-kube-api-access-f67j8\") pod \"payload-processing-e2e-mt-65f07c0f-7f7d7bf96f-5b7l2\" (UID: \"6aa65db8-d977-426d-8336-aa870a95fb4c\") " pod="openshift-ingress/payload-processing-e2e-mt-65f07c0f-7f7d7bf96f-5b7l2" Sep 07 16:35:03.510835 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:03.510660 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6aa65db8-d977-426d-8336-aa870a95fb4c-plugins-config-volume\") pod \"payload-processing-e2e-mt-65f07c0f-7f7d7bf96f-5b7l2\" (UID: \"6aa65db8-d977-426d-8336-aa870a95fb4c\") " pod="openshift-ingress/payload-processing-e2e-mt-65f07c0f-7f7d7bf96f-5b7l2" Sep 07 16:35:03.510906 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:03.510892 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7279z\" (UniqueName: \"kubernetes.io/projected/79ac266c-6176-4cb9-8486-05f777b73bc9-kube-api-access-7279z\") pod \"payload-pre-processing-e2e-mt-65f07c0f-6ffc48f886-gmprf\" (UID: \"79ac266c-6176-4cb9-8486-05f777b73bc9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-65f07c0f-6ffc48f886-gmprf" Sep 07 16:35:03.511067 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:03.510989 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79ac266c-6176-4cb9-8486-05f777b73bc9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-65f07c0f-6ffc48f886-gmprf\" (UID: \"79ac266c-6176-4cb9-8486-05f777b73bc9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-65f07c0f-6ffc48f886-gmprf" Sep 07 16:35:03.511533 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:03.511514 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6aa65db8-d977-426d-8336-aa870a95fb4c-plugins-config-volume\") pod \"payload-processing-e2e-mt-65f07c0f-7f7d7bf96f-5b7l2\" (UID: \"6aa65db8-d977-426d-8336-aa870a95fb4c\") " pod="openshift-ingress/payload-processing-e2e-mt-65f07c0f-7f7d7bf96f-5b7l2" Sep 07 16:35:03.511886 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:03.511867 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79ac266c-6176-4cb9-8486-05f777b73bc9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-65f07c0f-6ffc48f886-gmprf\" (UID: \"79ac266c-6176-4cb9-8486-05f777b73bc9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-65f07c0f-6ffc48f886-gmprf" Sep 07 16:35:03.518411 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:03.518389 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f67j8\" (UniqueName: \"kubernetes.io/projected/6aa65db8-d977-426d-8336-aa870a95fb4c-kube-api-access-f67j8\") pod \"payload-processing-e2e-mt-65f07c0f-7f7d7bf96f-5b7l2\" (UID: \"6aa65db8-d977-426d-8336-aa870a95fb4c\") " pod="openshift-ingress/payload-processing-e2e-mt-65f07c0f-7f7d7bf96f-5b7l2" Sep 07 16:35:03.518515 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:03.518388 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7279z\" (UniqueName: \"kubernetes.io/projected/79ac266c-6176-4cb9-8486-05f777b73bc9-kube-api-access-7279z\") pod \"payload-pre-processing-e2e-mt-65f07c0f-6ffc48f886-gmprf\" (UID: \"79ac266c-6176-4cb9-8486-05f777b73bc9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-65f07c0f-6ffc48f886-gmprf" Sep 07 16:35:03.552910 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:03.552871 2580 generic.go:356] "Generic (PLEG): container finished" podID="4aac6863-1edf-444d-a9d9-673bc3e80b82" containerID="575e5bfb10a5e8002a08a1f4d2ba7c0ee897cdf6174ea0babba278148a050919" exitCode=0 Sep 07 16:35:03.553147 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:03.552938 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw" Sep 07 16:35:03.553147 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:03.552964 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw" event={"ID":"4aac6863-1edf-444d-a9d9-673bc3e80b82","Type":"ContainerDied","Data":"575e5bfb10a5e8002a08a1f4d2ba7c0ee897cdf6174ea0babba278148a050919"} Sep 07 16:35:03.553147 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:03.553051 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw" event={"ID":"4aac6863-1edf-444d-a9d9-673bc3e80b82","Type":"ContainerDied","Data":"d73a9fa8fedbd474ffa943a540861cb0c0ed077cc4212ad97138d3a0c0294a45"} Sep 07 16:35:03.553147 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:03.553085 2580 scope.go:117] "RemoveContainer" containerID="575e5bfb10a5e8002a08a1f4d2ba7c0ee897cdf6174ea0babba278148a050919" Sep 07 16:35:03.564727 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:03.564703 2580 scope.go:117] "RemoveContainer" containerID="575e5bfb10a5e8002a08a1f4d2ba7c0ee897cdf6174ea0babba278148a050919" Sep 07 16:35:03.565179 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:35:03.565153 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"575e5bfb10a5e8002a08a1f4d2ba7c0ee897cdf6174ea0babba278148a050919\": container with ID starting with 575e5bfb10a5e8002a08a1f4d2ba7c0ee897cdf6174ea0babba278148a050919 not found: ID does not exist" containerID="575e5bfb10a5e8002a08a1f4d2ba7c0ee897cdf6174ea0babba278148a050919" Sep 07 16:35:03.565312 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:03.565184 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"575e5bfb10a5e8002a08a1f4d2ba7c0ee897cdf6174ea0babba278148a050919"} err="failed to get container status \"575e5bfb10a5e8002a08a1f4d2ba7c0ee897cdf6174ea0babba278148a050919\": rpc error: code = NotFound desc = could not find container \"575e5bfb10a5e8002a08a1f4d2ba7c0ee897cdf6174ea0babba278148a050919\": container with ID starting with 575e5bfb10a5e8002a08a1f4d2ba7c0ee897cdf6174ea0babba278148a050919 not found: ID does not exist" Sep 07 16:35:03.570417 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:03.570381 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw"] Sep 07 16:35:03.573884 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:03.573834 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-8a18f8-openshift-default-78f8fc5668-2qjrw"] Sep 07 16:35:03.659555 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:03.659458 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-65f07c0f-6ffc48f886-gmprf" Sep 07 16:35:03.679057 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:03.679007 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-65f07c0f-7f7d7bf96f-5b7l2" Sep 07 16:35:03.940197 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:03.940112 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-65f07c0f-6ffc48f886-gmprf"] Sep 07 16:35:03.943111 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:35:03.943082 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79ac266c_6176_4cb9_8486_05f777b73bc9.slice/crio-85eae4f7e1e22dab2db0f7ed92382385e980a5f08a6b527ec2639cf6b0389d04 WatchSource:0}: Error finding container 85eae4f7e1e22dab2db0f7ed92382385e980a5f08a6b527ec2639cf6b0389d04: Status 404 returned error can't find the container with id 85eae4f7e1e22dab2db0f7ed92382385e980a5f08a6b527ec2639cf6b0389d04 Sep 07 16:35:03.965997 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:03.965971 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-65f07c0f-7f7d7bf96f-5b7l2"] Sep 07 16:35:03.983220 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:35:03.983185 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aa65db8_d977_426d_8336_aa870a95fb4c.slice/crio-d5dc325287f95fe70f58be834807d26306611f96d8a44facbfc7c96a43b6be8e WatchSource:0}: Error finding container d5dc325287f95fe70f58be834807d26306611f96d8a44facbfc7c96a43b6be8e: Status 404 returned error can't find the container with id d5dc325287f95fe70f58be834807d26306611f96d8a44facbfc7c96a43b6be8e Sep 07 16:35:04.563985 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:04.563944 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-65f07c0f-6ffc48f886-gmprf" event={"ID":"79ac266c-6176-4cb9-8486-05f777b73bc9","Type":"ContainerStarted","Data":"6ce6ae9c50d174171333d3a3e130c5a5d097446b2cc77bc07ff145c0c7c9d343"} Sep 07 16:35:04.564209 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:04.564014 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-65f07c0f-6ffc48f886-gmprf" event={"ID":"79ac266c-6176-4cb9-8486-05f777b73bc9","Type":"ContainerStarted","Data":"85eae4f7e1e22dab2db0f7ed92382385e980a5f08a6b527ec2639cf6b0389d04"} Sep 07 16:35:04.564209 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:04.564062 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-65f07c0f-6ffc48f886-gmprf" Sep 07 16:35:04.565694 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:04.565669 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-65f07c0f-7f7d7bf96f-5b7l2" event={"ID":"6aa65db8-d977-426d-8336-aa870a95fb4c","Type":"ContainerStarted","Data":"a0361f3d0554c6422fa38ffb61fc992b8505a1240a478794f040782348a52d03"} Sep 07 16:35:04.565802 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:04.565713 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-65f07c0f-7f7d7bf96f-5b7l2" event={"ID":"6aa65db8-d977-426d-8336-aa870a95fb4c","Type":"ContainerStarted","Data":"d5dc325287f95fe70f58be834807d26306611f96d8a44facbfc7c96a43b6be8e"} Sep 07 16:35:04.565880 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:04.565805 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-65f07c0f-7f7d7bf96f-5b7l2" Sep 07 16:35:04.583165 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:04.583103 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-65f07c0f-6ffc48f886-gmprf" podStartSLOduration=1.5830801129999998 podStartE2EDuration="1.583080113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:35:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:35:04.581488416 +0000 UTC m=+933.616711489" watchObservedRunningTime="2026-09-07 16:35:04.583080113 +0000 UTC m=+933.618303186" Sep 07 16:35:04.596219 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:04.596150 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-65f07c0f-7f7d7bf96f-5b7l2" podStartSLOduration=1.596128872 podStartE2EDuration="1.596128872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:35:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:35:04.595375045 +0000 UTC m=+933.630598116" watchObservedRunningTime="2026-09-07 16:35:04.596128872 +0000 UTC m=+933.631351943" Sep 07 16:35:05.418391 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:05.418360 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4aac6863-1edf-444d-a9d9-673bc3e80b82" path="/var/lib/kubelet/pods/4aac6863-1edf-444d-a9d9-673bc3e80b82/volumes" Sep 07 16:35:09.140226 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:09.140192 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-a7d399/test-model-a7d399-kserve-56f8498599-vwp4s"] Sep 07 16:35:15.573089 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:15.573052 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-65f07c0f-7f7d7bf96f-5b7l2" Sep 07 16:35:15.573536 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:15.573129 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-65f07c0f-6ffc48f886-gmprf" Sep 07 16:35:19.751150 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:19.750079 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-747d6d-8d79cd766-hnpxc"] Sep 07 16:35:19.751150 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:19.750913 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-747d6d-8d79cd766-hnpxc" podUID="491162f6-cac0-42a6-b71e-1c9d976f95f9" containerName="payload-pre-processing" containerID="cri-o://686fdd8cc88cf1e12e2201310cf2cfc548655f4a30aa2f8b5f5715555153d006" gracePeriod=30 Sep 07 16:35:20.027746 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:20.027721 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-747d6d-8d79cd766-hnpxc" Sep 07 16:35:20.178314 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:20.178258 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q4g4v\" (UniqueName: \"kubernetes.io/projected/491162f6-cac0-42a6-b71e-1c9d976f95f9-kube-api-access-q4g4v\") pod \"491162f6-cac0-42a6-b71e-1c9d976f95f9\" (UID: \"491162f6-cac0-42a6-b71e-1c9d976f95f9\") " Sep 07 16:35:20.178468 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:20.178452 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/491162f6-cac0-42a6-b71e-1c9d976f95f9-plugins-config-volume\") pod \"491162f6-cac0-42a6-b71e-1c9d976f95f9\" (UID: \"491162f6-cac0-42a6-b71e-1c9d976f95f9\") " Sep 07 16:35:20.178813 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:20.178779 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/491162f6-cac0-42a6-b71e-1c9d976f95f9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "491162f6-cac0-42a6-b71e-1c9d976f95f9" (UID: "491162f6-cac0-42a6-b71e-1c9d976f95f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 16:35:20.180788 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:20.180766 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/491162f6-cac0-42a6-b71e-1c9d976f95f9-kube-api-access-q4g4v" (OuterVolumeSpecName: "kube-api-access-q4g4v") pod "491162f6-cac0-42a6-b71e-1c9d976f95f9" (UID: "491162f6-cac0-42a6-b71e-1c9d976f95f9"). InnerVolumeSpecName "kube-api-access-q4g4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 16:35:20.279239 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:20.279145 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/491162f6-cac0-42a6-b71e-1c9d976f95f9-plugins-config-volume\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:35:20.279239 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:20.279179 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4g4v\" (UniqueName: \"kubernetes.io/projected/491162f6-cac0-42a6-b71e-1c9d976f95f9-kube-api-access-q4g4v\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:35:20.660548 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:20.660467 2580 generic.go:356] "Generic (PLEG): container finished" podID="491162f6-cac0-42a6-b71e-1c9d976f95f9" containerID="686fdd8cc88cf1e12e2201310cf2cfc548655f4a30aa2f8b5f5715555153d006" exitCode=0 Sep 07 16:35:20.660548 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:20.660534 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-747d6d-8d79cd766-hnpxc" Sep 07 16:35:20.660741 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:20.660558 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-747d6d-8d79cd766-hnpxc" event={"ID":"491162f6-cac0-42a6-b71e-1c9d976f95f9","Type":"ContainerDied","Data":"686fdd8cc88cf1e12e2201310cf2cfc548655f4a30aa2f8b5f5715555153d006"} Sep 07 16:35:20.660741 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:20.660623 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-747d6d-8d79cd766-hnpxc" event={"ID":"491162f6-cac0-42a6-b71e-1c9d976f95f9","Type":"ContainerDied","Data":"d6f13e202dae9ec38fd244d5979a0a497e13bf1aefeed4219659135f6ce76d07"} Sep 07 16:35:20.660741 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:20.660645 2580 scope.go:117] "RemoveContainer" containerID="686fdd8cc88cf1e12e2201310cf2cfc548655f4a30aa2f8b5f5715555153d006" Sep 07 16:35:20.670817 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:20.670785 2580 scope.go:117] "RemoveContainer" containerID="686fdd8cc88cf1e12e2201310cf2cfc548655f4a30aa2f8b5f5715555153d006" Sep 07 16:35:20.671491 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:35:20.671449 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"686fdd8cc88cf1e12e2201310cf2cfc548655f4a30aa2f8b5f5715555153d006\": container with ID starting with 686fdd8cc88cf1e12e2201310cf2cfc548655f4a30aa2f8b5f5715555153d006 not found: ID does not exist" containerID="686fdd8cc88cf1e12e2201310cf2cfc548655f4a30aa2f8b5f5715555153d006" Sep 07 16:35:20.671622 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:20.671491 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"686fdd8cc88cf1e12e2201310cf2cfc548655f4a30aa2f8b5f5715555153d006"} err="failed to get container status \"686fdd8cc88cf1e12e2201310cf2cfc548655f4a30aa2f8b5f5715555153d006\": rpc error: code = NotFound desc = could not find container \"686fdd8cc88cf1e12e2201310cf2cfc548655f4a30aa2f8b5f5715555153d006\": container with ID starting with 686fdd8cc88cf1e12e2201310cf2cfc548655f4a30aa2f8b5f5715555153d006 not found: ID does not exist" Sep 07 16:35:20.684879 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:20.684844 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-747d6d-8d79cd766-hnpxc"] Sep 07 16:35:20.688789 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:20.688763 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-747d6d-8d79cd766-hnpxc"] Sep 07 16:35:21.418966 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:21.418923 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="491162f6-cac0-42a6-b71e-1c9d976f95f9" path="/var/lib/kubelet/pods/491162f6-cac0-42a6-b71e-1c9d976f95f9/volumes" Sep 07 16:35:22.678375 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:22.678257 2580 generic.go:356] "Generic (PLEG): container finished" podID="275283fc-2837-4fe0-884b-95803aac56c4" containerID="a60850c31e11e979bdb5c12235b8d4a4828ecc94fe465ad39ec71505615cc201" exitCode=0 Sep 07 16:35:22.678375 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:22.678337 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm" event={"ID":"275283fc-2837-4fe0-884b-95803aac56c4","Type":"ContainerDied","Data":"a60850c31e11e979bdb5c12235b8d4a4828ecc94fe465ad39ec71505615cc201"} Sep 07 16:35:22.678944 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:22.678854 2580 scope.go:117] "RemoveContainer" containerID="a60850c31e11e979bdb5c12235b8d4a4828ecc94fe465ad39ec71505615cc201" Sep 07 16:35:22.679998 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:22.679969 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:35:22.680094 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:22.680076 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:35:22.680133 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:22.680125 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:35:23.685221 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:23.685190 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm" event={"ID":"275283fc-2837-4fe0-884b-95803aac56c4","Type":"ContainerStarted","Data":"43cd6ab41de21d583e5d91a5730f52fec3c92db3837add4197b3b1ebafd7ba50"} Sep 07 16:35:24.161049 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:24.160981 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-a7d399/test-model-a7d399-kserve-56f8498599-vwp4s" podUID="65552f5b-548e-4df7-a581-eae11be6962e" containerName="main" containerID="cri-o://df7a221086e9206652d9048fef220bc1d9bf8f0e82d5d08def1e57e358042c7f" gracePeriod=45 Sep 07 16:35:24.457412 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:24.457385 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a7d399/test-model-a7d399-kserve-56f8498599-vwp4s" Sep 07 16:35:24.585623 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:24.585584 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm" Sep 07 16:35:24.585812 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:24.585784 2580 patch_prober.go:28] interesting pod/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Sep 07 16:35:24.585859 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:24.585832 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm" podUID="275283fc-2837-4fe0-884b-95803aac56c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Sep 07 16:35:24.635056 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:24.633310 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s68c9\" (UniqueName: \"kubernetes.io/projected/65552f5b-548e-4df7-a581-eae11be6962e-kube-api-access-s68c9\") pod \"65552f5b-548e-4df7-a581-eae11be6962e\" (UID: \"65552f5b-548e-4df7-a581-eae11be6962e\") " Sep 07 16:35:24.635056 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:24.633402 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65552f5b-548e-4df7-a581-eae11be6962e-model-cache\") pod \"65552f5b-548e-4df7-a581-eae11be6962e\" (UID: \"65552f5b-548e-4df7-a581-eae11be6962e\") " Sep 07 16:35:24.635056 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:24.633448 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65552f5b-548e-4df7-a581-eae11be6962e-home\") pod \"65552f5b-548e-4df7-a581-eae11be6962e\" (UID: \"65552f5b-548e-4df7-a581-eae11be6962e\") " Sep 07 16:35:24.635056 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:24.633499 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65552f5b-548e-4df7-a581-eae11be6962e-tls-certs\") pod \"65552f5b-548e-4df7-a581-eae11be6962e\" (UID: \"65552f5b-548e-4df7-a581-eae11be6962e\") " Sep 07 16:35:24.635056 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:24.633546 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65552f5b-548e-4df7-a581-eae11be6962e-dshm\") pod \"65552f5b-548e-4df7-a581-eae11be6962e\" (UID: \"65552f5b-548e-4df7-a581-eae11be6962e\") " Sep 07 16:35:24.635056 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:24.633581 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65552f5b-548e-4df7-a581-eae11be6962e-tmp-dir\") pod \"65552f5b-548e-4df7-a581-eae11be6962e\" (UID: \"65552f5b-548e-4df7-a581-eae11be6962e\") " Sep 07 16:35:24.635056 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:24.634145 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65552f5b-548e-4df7-a581-eae11be6962e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "65552f5b-548e-4df7-a581-eae11be6962e" (UID: "65552f5b-548e-4df7-a581-eae11be6962e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 16:35:24.635535 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:24.635242 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65552f5b-548e-4df7-a581-eae11be6962e-home" (OuterVolumeSpecName: "home") pod "65552f5b-548e-4df7-a581-eae11be6962e" (UID: "65552f5b-548e-4df7-a581-eae11be6962e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 16:35:24.635535 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:24.635317 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65552f5b-548e-4df7-a581-eae11be6962e-model-cache" (OuterVolumeSpecName: "model-cache") pod "65552f5b-548e-4df7-a581-eae11be6962e" (UID: "65552f5b-548e-4df7-a581-eae11be6962e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 16:35:24.637868 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:24.637806 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65552f5b-548e-4df7-a581-eae11be6962e-kube-api-access-s68c9" (OuterVolumeSpecName: "kube-api-access-s68c9") pod "65552f5b-548e-4df7-a581-eae11be6962e" (UID: "65552f5b-548e-4df7-a581-eae11be6962e"). InnerVolumeSpecName "kube-api-access-s68c9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 16:35:24.638115 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:24.638087 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65552f5b-548e-4df7-a581-eae11be6962e-dshm" (OuterVolumeSpecName: "dshm") pod "65552f5b-548e-4df7-a581-eae11be6962e" (UID: "65552f5b-548e-4df7-a581-eae11be6962e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 16:35:24.638216 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:24.638167 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65552f5b-548e-4df7-a581-eae11be6962e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "65552f5b-548e-4df7-a581-eae11be6962e" (UID: "65552f5b-548e-4df7-a581-eae11be6962e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 16:35:24.690762 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:24.690667 2580 generic.go:356] "Generic (PLEG): container finished" podID="65552f5b-548e-4df7-a581-eae11be6962e" containerID="df7a221086e9206652d9048fef220bc1d9bf8f0e82d5d08def1e57e358042c7f" exitCode=0 Sep 07 16:35:24.690762 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:24.690736 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a7d399/test-model-a7d399-kserve-56f8498599-vwp4s" Sep 07 16:35:24.691284 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:24.690757 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a7d399/test-model-a7d399-kserve-56f8498599-vwp4s" event={"ID":"65552f5b-548e-4df7-a581-eae11be6962e","Type":"ContainerDied","Data":"df7a221086e9206652d9048fef220bc1d9bf8f0e82d5d08def1e57e358042c7f"} Sep 07 16:35:24.691284 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:24.690801 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a7d399/test-model-a7d399-kserve-56f8498599-vwp4s" event={"ID":"65552f5b-548e-4df7-a581-eae11be6962e","Type":"ContainerDied","Data":"ddd7aca7c13c165e57ba99902c30f99590708706c624323a0063421f1268938f"} Sep 07 16:35:24.691284 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:24.690823 2580 scope.go:117] "RemoveContainer" containerID="df7a221086e9206652d9048fef220bc1d9bf8f0e82d5d08def1e57e358042c7f" Sep 07 16:35:24.699892 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:24.699875 2580 scope.go:117] "RemoveContainer" containerID="df7a221086e9206652d9048fef220bc1d9bf8f0e82d5d08def1e57e358042c7f" Sep 07 16:35:24.700205 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:35:24.700187 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df7a221086e9206652d9048fef220bc1d9bf8f0e82d5d08def1e57e358042c7f\": container with ID starting with df7a221086e9206652d9048fef220bc1d9bf8f0e82d5d08def1e57e358042c7f not found: ID does not exist" containerID="df7a221086e9206652d9048fef220bc1d9bf8f0e82d5d08def1e57e358042c7f" Sep 07 16:35:24.700256 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:24.700215 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df7a221086e9206652d9048fef220bc1d9bf8f0e82d5d08def1e57e358042c7f"} err="failed to get container status \"df7a221086e9206652d9048fef220bc1d9bf8f0e82d5d08def1e57e358042c7f\": rpc error: code = NotFound desc = could not find container \"df7a221086e9206652d9048fef220bc1d9bf8f0e82d5d08def1e57e358042c7f\": container with ID starting with df7a221086e9206652d9048fef220bc1d9bf8f0e82d5d08def1e57e358042c7f not found: ID does not exist" Sep 07 16:35:24.723050 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:24.710636 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-a7d399/test-model-a7d399-kserve-56f8498599-vwp4s"] Sep 07 16:35:24.734369 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:24.734323 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-a7d399/test-model-a7d399-kserve-56f8498599-vwp4s"] Sep 07 16:35:24.735444 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:24.735411 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65552f5b-548e-4df7-a581-eae11be6962e-tls-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:35:24.735444 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:24.735440 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65552f5b-548e-4df7-a581-eae11be6962e-dshm\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:35:24.735444 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:24.735448 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65552f5b-548e-4df7-a581-eae11be6962e-tmp-dir\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:35:24.735710 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:24.735456 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s68c9\" (UniqueName: \"kubernetes.io/projected/65552f5b-548e-4df7-a581-eae11be6962e-kube-api-access-s68c9\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:35:24.735710 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:24.735466 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65552f5b-548e-4df7-a581-eae11be6962e-model-cache\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:35:24.735710 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:24.735473 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65552f5b-548e-4df7-a581-eae11be6962e-home\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:35:25.438108 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:25.438066 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65552f5b-548e-4df7-a581-eae11be6962e" path="/var/lib/kubelet/pods/65552f5b-548e-4df7-a581-eae11be6962e/volumes" Sep 07 16:35:25.586119 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:25.586083 2580 patch_prober.go:28] interesting pod/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Sep 07 16:35:25.586306 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:25.586143 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm" podUID="275283fc-2837-4fe0-884b-95803aac56c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Sep 07 16:35:26.585872 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:26.585833 2580 patch_prober.go:28] interesting pod/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Sep 07 16:35:26.586317 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:26.585896 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm" podUID="275283fc-2837-4fe0-884b-95803aac56c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Sep 07 16:35:27.586081 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:27.586000 2580 patch_prober.go:28] interesting pod/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Sep 07 16:35:27.586625 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:27.586124 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm" podUID="275283fc-2837-4fe0-884b-95803aac56c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Sep 07 16:35:28.585600 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:28.585555 2580 patch_prober.go:28] interesting pod/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Sep 07 16:35:28.585786 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:28.585626 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm" podUID="275283fc-2837-4fe0-884b-95803aac56c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Sep 07 16:35:29.624728 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:29.624698 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm" Sep 07 16:35:29.625159 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:29.625065 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm" Sep 07 16:35:29.626136 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:29.626116 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm" Sep 07 16:35:44.453474 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:44.453431 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-65f07c0f-7f7d7bf96f-5b7l2"] Sep 07 16:35:44.454052 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:44.453720 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-65f07c0f-7f7d7bf96f-5b7l2" podUID="6aa65db8-d977-426d-8336-aa870a95fb4c" containerName="payload-processing" containerID="cri-o://a0361f3d0554c6422fa38ffb61fc992b8505a1240a478794f040782348a52d03" gracePeriod=30 Sep 07 16:35:44.463917 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:44.463887 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-65f07c0f-6ffc48f886-gmprf"] Sep 07 16:35:44.464192 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:44.464166 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-65f07c0f-6ffc48f886-gmprf" podUID="79ac266c-6176-4cb9-8486-05f777b73bc9" containerName="payload-pre-processing" containerID="cri-o://6ce6ae9c50d174171333d3a3e130c5a5d097446b2cc77bc07ff145c0c7c9d343" gracePeriod=30 Sep 07 16:35:44.713158 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:44.713092 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-65f07c0f-7f7d7bf96f-5b7l2" Sep 07 16:35:44.744702 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:44.744676 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-65f07c0f-6ffc48f886-gmprf" Sep 07 16:35:44.791561 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:44.791528 2580 generic.go:356] "Generic (PLEG): container finished" podID="79ac266c-6176-4cb9-8486-05f777b73bc9" containerID="6ce6ae9c50d174171333d3a3e130c5a5d097446b2cc77bc07ff145c0c7c9d343" exitCode=0 Sep 07 16:35:44.791561 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:44.791567 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-65f07c0f-6ffc48f886-gmprf" Sep 07 16:35:44.791820 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:44.791610 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-65f07c0f-6ffc48f886-gmprf" event={"ID":"79ac266c-6176-4cb9-8486-05f777b73bc9","Type":"ContainerDied","Data":"6ce6ae9c50d174171333d3a3e130c5a5d097446b2cc77bc07ff145c0c7c9d343"} Sep 07 16:35:44.791820 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:44.791674 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-65f07c0f-6ffc48f886-gmprf" event={"ID":"79ac266c-6176-4cb9-8486-05f777b73bc9","Type":"ContainerDied","Data":"85eae4f7e1e22dab2db0f7ed92382385e980a5f08a6b527ec2639cf6b0389d04"} Sep 07 16:35:44.791820 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:44.791701 2580 scope.go:117] "RemoveContainer" containerID="6ce6ae9c50d174171333d3a3e130c5a5d097446b2cc77bc07ff145c0c7c9d343" Sep 07 16:35:44.793135 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:44.793112 2580 generic.go:356] "Generic (PLEG): container finished" podID="6aa65db8-d977-426d-8336-aa870a95fb4c" containerID="a0361f3d0554c6422fa38ffb61fc992b8505a1240a478794f040782348a52d03" exitCode=0 Sep 07 16:35:44.793250 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:44.793144 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-65f07c0f-7f7d7bf96f-5b7l2" event={"ID":"6aa65db8-d977-426d-8336-aa870a95fb4c","Type":"ContainerDied","Data":"a0361f3d0554c6422fa38ffb61fc992b8505a1240a478794f040782348a52d03"} Sep 07 16:35:44.793250 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:44.793162 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-65f07c0f-7f7d7bf96f-5b7l2" Sep 07 16:35:44.793250 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:44.793193 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-65f07c0f-7f7d7bf96f-5b7l2" event={"ID":"6aa65db8-d977-426d-8336-aa870a95fb4c","Type":"ContainerDied","Data":"d5dc325287f95fe70f58be834807d26306611f96d8a44facbfc7c96a43b6be8e"} Sep 07 16:35:44.801572 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:44.801555 2580 scope.go:117] "RemoveContainer" containerID="6ce6ae9c50d174171333d3a3e130c5a5d097446b2cc77bc07ff145c0c7c9d343" Sep 07 16:35:44.801931 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:35:44.801904 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ce6ae9c50d174171333d3a3e130c5a5d097446b2cc77bc07ff145c0c7c9d343\": container with ID starting with 6ce6ae9c50d174171333d3a3e130c5a5d097446b2cc77bc07ff145c0c7c9d343 not found: ID does not exist" containerID="6ce6ae9c50d174171333d3a3e130c5a5d097446b2cc77bc07ff145c0c7c9d343" Sep 07 16:35:44.801997 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:44.801943 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ce6ae9c50d174171333d3a3e130c5a5d097446b2cc77bc07ff145c0c7c9d343"} err="failed to get container status \"6ce6ae9c50d174171333d3a3e130c5a5d097446b2cc77bc07ff145c0c7c9d343\": rpc error: code = NotFound desc = could not find container \"6ce6ae9c50d174171333d3a3e130c5a5d097446b2cc77bc07ff145c0c7c9d343\": container with ID starting with 6ce6ae9c50d174171333d3a3e130c5a5d097446b2cc77bc07ff145c0c7c9d343 not found: ID does not exist" Sep 07 16:35:44.801997 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:44.801964 2580 scope.go:117] "RemoveContainer" containerID="a0361f3d0554c6422fa38ffb61fc992b8505a1240a478794f040782348a52d03" Sep 07 16:35:44.810687 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:44.810664 2580 scope.go:117] "RemoveContainer" containerID="a0361f3d0554c6422fa38ffb61fc992b8505a1240a478794f040782348a52d03" Sep 07 16:35:44.811073 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:35:44.811051 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0361f3d0554c6422fa38ffb61fc992b8505a1240a478794f040782348a52d03\": container with ID starting with a0361f3d0554c6422fa38ffb61fc992b8505a1240a478794f040782348a52d03 not found: ID does not exist" containerID="a0361f3d0554c6422fa38ffb61fc992b8505a1240a478794f040782348a52d03" Sep 07 16:35:44.811177 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:44.811087 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0361f3d0554c6422fa38ffb61fc992b8505a1240a478794f040782348a52d03"} err="failed to get container status \"a0361f3d0554c6422fa38ffb61fc992b8505a1240a478794f040782348a52d03\": rpc error: code = NotFound desc = could not find container \"a0361f3d0554c6422fa38ffb61fc992b8505a1240a478794f040782348a52d03\": container with ID starting with a0361f3d0554c6422fa38ffb61fc992b8505a1240a478794f040782348a52d03 not found: ID does not exist" Sep 07 16:35:44.823506 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:44.823480 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7279z\" (UniqueName: \"kubernetes.io/projected/79ac266c-6176-4cb9-8486-05f777b73bc9-kube-api-access-7279z\") pod \"79ac266c-6176-4cb9-8486-05f777b73bc9\" (UID: \"79ac266c-6176-4cb9-8486-05f777b73bc9\") " Sep 07 16:35:44.823651 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:44.823546 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79ac266c-6176-4cb9-8486-05f777b73bc9-plugins-config-volume\") pod \"79ac266c-6176-4cb9-8486-05f777b73bc9\" (UID: \"79ac266c-6176-4cb9-8486-05f777b73bc9\") " Sep 07 16:35:44.823651 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:44.823565 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f67j8\" (UniqueName: \"kubernetes.io/projected/6aa65db8-d977-426d-8336-aa870a95fb4c-kube-api-access-f67j8\") pod \"6aa65db8-d977-426d-8336-aa870a95fb4c\" (UID: \"6aa65db8-d977-426d-8336-aa870a95fb4c\") " Sep 07 16:35:44.823651 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:44.823589 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6aa65db8-d977-426d-8336-aa870a95fb4c-plugins-config-volume\") pod \"6aa65db8-d977-426d-8336-aa870a95fb4c\" (UID: \"6aa65db8-d977-426d-8336-aa870a95fb4c\") " Sep 07 16:35:44.823998 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:44.823962 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79ac266c-6176-4cb9-8486-05f777b73bc9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "79ac266c-6176-4cb9-8486-05f777b73bc9" (UID: "79ac266c-6176-4cb9-8486-05f777b73bc9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 16:35:44.824114 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:44.824017 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aa65db8-d977-426d-8336-aa870a95fb4c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6aa65db8-d977-426d-8336-aa870a95fb4c" (UID: "6aa65db8-d977-426d-8336-aa870a95fb4c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 16:35:44.825935 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:44.825915 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aa65db8-d977-426d-8336-aa870a95fb4c-kube-api-access-f67j8" (OuterVolumeSpecName: "kube-api-access-f67j8") pod "6aa65db8-d977-426d-8336-aa870a95fb4c" (UID: "6aa65db8-d977-426d-8336-aa870a95fb4c"). InnerVolumeSpecName "kube-api-access-f67j8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 16:35:44.825993 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:44.825933 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79ac266c-6176-4cb9-8486-05f777b73bc9-kube-api-access-7279z" (OuterVolumeSpecName: "kube-api-access-7279z") pod "79ac266c-6176-4cb9-8486-05f777b73bc9" (UID: "79ac266c-6176-4cb9-8486-05f777b73bc9"). InnerVolumeSpecName "kube-api-access-7279z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 16:35:44.924938 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:44.924905 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79ac266c-6176-4cb9-8486-05f777b73bc9-plugins-config-volume\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:35:44.924938 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:44.924933 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f67j8\" (UniqueName: \"kubernetes.io/projected/6aa65db8-d977-426d-8336-aa870a95fb4c-kube-api-access-f67j8\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:35:44.924938 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:44.924943 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6aa65db8-d977-426d-8336-aa870a95fb4c-plugins-config-volume\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:35:44.925272 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:44.924952 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7279z\" (UniqueName: \"kubernetes.io/projected/79ac266c-6176-4cb9-8486-05f777b73bc9-kube-api-access-7279z\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:35:45.115459 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:45.115427 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-65f07c0f-6ffc48f886-gmprf"] Sep 07 16:35:45.118798 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:45.118774 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-65f07c0f-6ffc48f886-gmprf"] Sep 07 16:35:45.128929 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:45.128903 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-65f07c0f-7f7d7bf96f-5b7l2"] Sep 07 16:35:45.132168 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:45.132146 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-65f07c0f-7f7d7bf96f-5b7l2"] Sep 07 16:35:45.418830 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:45.418731 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6aa65db8-d977-426d-8336-aa870a95fb4c" path="/var/lib/kubelet/pods/6aa65db8-d977-426d-8336-aa870a95fb4c/volumes" Sep 07 16:35:45.419244 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:45.419221 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79ac266c-6176-4cb9-8486-05f777b73bc9" path="/var/lib/kubelet/pods/79ac266c-6176-4cb9-8486-05f777b73bc9/volumes" Sep 07 16:35:51.116010 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:51.115973 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm"] Sep 07 16:35:51.116507 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:51.116306 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm" podUID="275283fc-2837-4fe0-884b-95803aac56c4" containerName="istio-proxy" containerID="cri-o://43cd6ab41de21d583e5d91a5730f52fec3c92db3837add4197b3b1ebafd7ba50" gracePeriod=30 Sep 07 16:35:56.365833 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:56.365803 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm" Sep 07 16:35:56.531935 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:56.531891 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/275283fc-2837-4fe0-884b-95803aac56c4-istiod-ca-cert\") pod \"275283fc-2837-4fe0-884b-95803aac56c4\" (UID: \"275283fc-2837-4fe0-884b-95803aac56c4\") " Sep 07 16:35:56.531935 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:56.531940 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/275283fc-2837-4fe0-884b-95803aac56c4-workload-socket\") pod \"275283fc-2837-4fe0-884b-95803aac56c4\" (UID: \"275283fc-2837-4fe0-884b-95803aac56c4\") " Sep 07 16:35:56.532179 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:56.531961 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/275283fc-2837-4fe0-884b-95803aac56c4-istio-token\") pod \"275283fc-2837-4fe0-884b-95803aac56c4\" (UID: \"275283fc-2837-4fe0-884b-95803aac56c4\") " Sep 07 16:35:56.532179 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:56.531986 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/275283fc-2837-4fe0-884b-95803aac56c4-workload-certs\") pod \"275283fc-2837-4fe0-884b-95803aac56c4\" (UID: \"275283fc-2837-4fe0-884b-95803aac56c4\") " Sep 07 16:35:56.532179 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:56.532067 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xflj4\" (UniqueName: \"kubernetes.io/projected/275283fc-2837-4fe0-884b-95803aac56c4-kube-api-access-xflj4\") pod \"275283fc-2837-4fe0-884b-95803aac56c4\" (UID: \"275283fc-2837-4fe0-884b-95803aac56c4\") " Sep 07 16:35:56.532179 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:56.532131 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/275283fc-2837-4fe0-884b-95803aac56c4-istio-envoy\") pod \"275283fc-2837-4fe0-884b-95803aac56c4\" (UID: \"275283fc-2837-4fe0-884b-95803aac56c4\") " Sep 07 16:35:56.532385 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:56.532180 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/275283fc-2837-4fe0-884b-95803aac56c4-credential-socket\") pod \"275283fc-2837-4fe0-884b-95803aac56c4\" (UID: \"275283fc-2837-4fe0-884b-95803aac56c4\") " Sep 07 16:35:56.532385 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:56.532247 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/275283fc-2837-4fe0-884b-95803aac56c4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "275283fc-2837-4fe0-884b-95803aac56c4" (UID: "275283fc-2837-4fe0-884b-95803aac56c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 16:35:56.532385 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:56.532297 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/275283fc-2837-4fe0-884b-95803aac56c4-istio-data\") pod \"275283fc-2837-4fe0-884b-95803aac56c4\" (UID: \"275283fc-2837-4fe0-884b-95803aac56c4\") " Sep 07 16:35:56.532385 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:56.532354 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/275283fc-2837-4fe0-884b-95803aac56c4-istio-podinfo\") pod \"275283fc-2837-4fe0-884b-95803aac56c4\" (UID: \"275283fc-2837-4fe0-884b-95803aac56c4\") " Sep 07 16:35:56.532596 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:56.532410 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/275283fc-2837-4fe0-884b-95803aac56c4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "275283fc-2837-4fe0-884b-95803aac56c4" (UID: "275283fc-2837-4fe0-884b-95803aac56c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 16:35:56.532596 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:56.532465 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/275283fc-2837-4fe0-884b-95803aac56c4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "275283fc-2837-4fe0-884b-95803aac56c4" (UID: "275283fc-2837-4fe0-884b-95803aac56c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 16:35:56.532596 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:56.532487 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/275283fc-2837-4fe0-884b-95803aac56c4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "275283fc-2837-4fe0-884b-95803aac56c4" (UID: "275283fc-2837-4fe0-884b-95803aac56c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 16:35:56.532710 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:56.532631 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/275283fc-2837-4fe0-884b-95803aac56c4-istio-data" (OuterVolumeSpecName: "istio-data") pod "275283fc-2837-4fe0-884b-95803aac56c4" (UID: "275283fc-2837-4fe0-884b-95803aac56c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 16:35:56.532756 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:56.532717 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/275283fc-2837-4fe0-884b-95803aac56c4-credential-socket\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:35:56.532756 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:56.532733 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/275283fc-2837-4fe0-884b-95803aac56c4-istio-data\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:35:56.532756 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:56.532745 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/275283fc-2837-4fe0-884b-95803aac56c4-istiod-ca-cert\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:35:56.532864 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:56.532757 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/275283fc-2837-4fe0-884b-95803aac56c4-workload-socket\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:35:56.532864 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:56.532772 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/275283fc-2837-4fe0-884b-95803aac56c4-workload-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:35:56.534631 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:56.534590 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/275283fc-2837-4fe0-884b-95803aac56c4-kube-api-access-xflj4" (OuterVolumeSpecName: "kube-api-access-xflj4") pod "275283fc-2837-4fe0-884b-95803aac56c4" (UID: "275283fc-2837-4fe0-884b-95803aac56c4"). InnerVolumeSpecName "kube-api-access-xflj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 16:35:56.534631 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:56.534605 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/275283fc-2837-4fe0-884b-95803aac56c4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "275283fc-2837-4fe0-884b-95803aac56c4" (UID: "275283fc-2837-4fe0-884b-95803aac56c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 07 16:35:56.534830 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:56.534813 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/275283fc-2837-4fe0-884b-95803aac56c4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "275283fc-2837-4fe0-884b-95803aac56c4" (UID: "275283fc-2837-4fe0-884b-95803aac56c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 16:35:56.534889 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:56.534833 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/275283fc-2837-4fe0-884b-95803aac56c4-istio-token" (OuterVolumeSpecName: "istio-token") pod "275283fc-2837-4fe0-884b-95803aac56c4" (UID: "275283fc-2837-4fe0-884b-95803aac56c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 16:35:56.634085 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:56.634007 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/275283fc-2837-4fe0-884b-95803aac56c4-istio-token\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:35:56.634085 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:56.634076 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xflj4\" (UniqueName: \"kubernetes.io/projected/275283fc-2837-4fe0-884b-95803aac56c4-kube-api-access-xflj4\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:35:56.634085 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:56.634087 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/275283fc-2837-4fe0-884b-95803aac56c4-istio-envoy\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:35:56.634085 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:56.634096 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/275283fc-2837-4fe0-884b-95803aac56c4-istio-podinfo\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:35:56.857608 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:56.857515 2580 generic.go:356] "Generic (PLEG): container finished" podID="275283fc-2837-4fe0-884b-95803aac56c4" containerID="43cd6ab41de21d583e5d91a5730f52fec3c92db3837add4197b3b1ebafd7ba50" exitCode=0 Sep 07 16:35:56.857608 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:56.857565 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm" Sep 07 16:35:56.857845 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:56.857566 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm" event={"ID":"275283fc-2837-4fe0-884b-95803aac56c4","Type":"ContainerDied","Data":"43cd6ab41de21d583e5d91a5730f52fec3c92db3837add4197b3b1ebafd7ba50"} Sep 07 16:35:56.857845 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:56.857710 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm" event={"ID":"275283fc-2837-4fe0-884b-95803aac56c4","Type":"ContainerDied","Data":"fe6bb6ccb4482df530cdb342bba0cd06ad1e42a005b201ce1181228142c0f80f"} Sep 07 16:35:56.857845 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:56.857752 2580 scope.go:117] "RemoveContainer" containerID="43cd6ab41de21d583e5d91a5730f52fec3c92db3837add4197b3b1ebafd7ba50" Sep 07 16:35:56.867362 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:56.867336 2580 scope.go:117] "RemoveContainer" containerID="a60850c31e11e979bdb5c12235b8d4a4828ecc94fe465ad39ec71505615cc201" Sep 07 16:35:56.875598 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:56.875574 2580 scope.go:117] "RemoveContainer" containerID="43cd6ab41de21d583e5d91a5730f52fec3c92db3837add4197b3b1ebafd7ba50" Sep 07 16:35:56.875991 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:35:56.875961 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43cd6ab41de21d583e5d91a5730f52fec3c92db3837add4197b3b1ebafd7ba50\": container with ID starting with 43cd6ab41de21d583e5d91a5730f52fec3c92db3837add4197b3b1ebafd7ba50 not found: ID does not exist" containerID="43cd6ab41de21d583e5d91a5730f52fec3c92db3837add4197b3b1ebafd7ba50" Sep 07 16:35:56.876134 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:56.876096 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43cd6ab41de21d583e5d91a5730f52fec3c92db3837add4197b3b1ebafd7ba50"} err="failed to get container status \"43cd6ab41de21d583e5d91a5730f52fec3c92db3837add4197b3b1ebafd7ba50\": rpc error: code = NotFound desc = could not find container \"43cd6ab41de21d583e5d91a5730f52fec3c92db3837add4197b3b1ebafd7ba50\": container with ID starting with 43cd6ab41de21d583e5d91a5730f52fec3c92db3837add4197b3b1ebafd7ba50 not found: ID does not exist" Sep 07 16:35:56.876134 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:56.876126 2580 scope.go:117] "RemoveContainer" containerID="a60850c31e11e979bdb5c12235b8d4a4828ecc94fe465ad39ec71505615cc201" Sep 07 16:35:56.876532 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:35:56.876492 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a60850c31e11e979bdb5c12235b8d4a4828ecc94fe465ad39ec71505615cc201\": container with ID starting with a60850c31e11e979bdb5c12235b8d4a4828ecc94fe465ad39ec71505615cc201 not found: ID does not exist" containerID="a60850c31e11e979bdb5c12235b8d4a4828ecc94fe465ad39ec71505615cc201" Sep 07 16:35:56.876611 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:56.876521 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a60850c31e11e979bdb5c12235b8d4a4828ecc94fe465ad39ec71505615cc201"} err="failed to get container status \"a60850c31e11e979bdb5c12235b8d4a4828ecc94fe465ad39ec71505615cc201\": rpc error: code = NotFound desc = could not find container \"a60850c31e11e979bdb5c12235b8d4a4828ecc94fe465ad39ec71505615cc201\": container with ID starting with a60850c31e11e979bdb5c12235b8d4a4828ecc94fe465ad39ec71505615cc201 not found: ID does not exist" Sep 07 16:35:56.877656 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:56.877636 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm"] Sep 07 16:35:56.881441 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:56.881420 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-65f07c0f-openshift-default-7bc4599dfb-4kssm"] Sep 07 16:35:57.417904 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:35:57.417871 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="275283fc-2837-4fe0-884b-95803aac56c4" path="/var/lib/kubelet/pods/275283fc-2837-4fe0-884b-95803aac56c4/volumes" Sep 07 16:36:10.957699 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:10.957663 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-bd6c5a-c65f7fb55-smjqr"] Sep 07 16:36:10.958133 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:10.957968 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="491162f6-cac0-42a6-b71e-1c9d976f95f9" containerName="payload-pre-processing" Sep 07 16:36:10.958133 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:10.957978 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="491162f6-cac0-42a6-b71e-1c9d976f95f9" containerName="payload-pre-processing" Sep 07 16:36:10.958133 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:10.957986 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="275283fc-2837-4fe0-884b-95803aac56c4" containerName="istio-proxy" Sep 07 16:36:10.958133 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:10.957992 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="275283fc-2837-4fe0-884b-95803aac56c4" containerName="istio-proxy" Sep 07 16:36:10.958133 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:10.958000 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65552f5b-548e-4df7-a581-eae11be6962e" containerName="main" Sep 07 16:36:10.958133 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:10.958005 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="65552f5b-548e-4df7-a581-eae11be6962e" containerName="main" Sep 07 16:36:10.958133 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:10.958014 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="275283fc-2837-4fe0-884b-95803aac56c4" containerName="istio-proxy" Sep 07 16:36:10.958133 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:10.958033 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="275283fc-2837-4fe0-884b-95803aac56c4" containerName="istio-proxy" Sep 07 16:36:10.958133 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:10.958040 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79ac266c-6176-4cb9-8486-05f777b73bc9" containerName="payload-pre-processing" Sep 07 16:36:10.958133 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:10.958046 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="79ac266c-6176-4cb9-8486-05f777b73bc9" containerName="payload-pre-processing" Sep 07 16:36:10.958133 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:10.958053 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6aa65db8-d977-426d-8336-aa870a95fb4c" containerName="payload-processing" Sep 07 16:36:10.958133 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:10.958059 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6aa65db8-d977-426d-8336-aa870a95fb4c" containerName="payload-processing" Sep 07 16:36:10.958133 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:10.958126 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="65552f5b-548e-4df7-a581-eae11be6962e" containerName="main" Sep 07 16:36:10.958133 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:10.958133 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="275283fc-2837-4fe0-884b-95803aac56c4" containerName="istio-proxy" Sep 07 16:36:10.958133 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:10.958139 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="275283fc-2837-4fe0-884b-95803aac56c4" containerName="istio-proxy" Sep 07 16:36:10.958809 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:10.958146 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="79ac266c-6176-4cb9-8486-05f777b73bc9" containerName="payload-pre-processing" Sep 07 16:36:10.958809 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:10.958151 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6aa65db8-d977-426d-8336-aa870a95fb4c" containerName="payload-processing" Sep 07 16:36:10.958809 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:10.958156 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="491162f6-cac0-42a6-b71e-1c9d976f95f9" containerName="payload-pre-processing" Sep 07 16:36:10.962093 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:10.962074 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bd6c5a-c65f7fb55-smjqr" Sep 07 16:36:10.964542 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:10.964515 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-bd6c5a-dockercfg-k9szt\"" Sep 07 16:36:10.964700 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:10.964516 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-bd6c5a\"" Sep 07 16:36:10.973906 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:10.973880 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-bd6c5a-c65f7fb55-smjqr"] Sep 07 16:36:11.063874 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:11.063828 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgbbd\" (UniqueName: \"kubernetes.io/projected/e6474d96-9621-43cd-bfd9-98079628793b-kube-api-access-fgbbd\") pod \"payload-processing-e2e-shared-a-w2-bd6c5a-c65f7fb55-smjqr\" (UID: \"e6474d96-9621-43cd-bfd9-98079628793b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bd6c5a-c65f7fb55-smjqr" Sep 07 16:36:11.063874 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:11.063878 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6474d96-9621-43cd-bfd9-98079628793b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-bd6c5a-c65f7fb55-smjqr\" (UID: \"e6474d96-9621-43cd-bfd9-98079628793b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bd6c5a-c65f7fb55-smjqr" Sep 07 16:36:11.164577 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:11.164538 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgbbd\" (UniqueName: \"kubernetes.io/projected/e6474d96-9621-43cd-bfd9-98079628793b-kube-api-access-fgbbd\") pod \"payload-processing-e2e-shared-a-w2-bd6c5a-c65f7fb55-smjqr\" (UID: \"e6474d96-9621-43cd-bfd9-98079628793b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bd6c5a-c65f7fb55-smjqr" Sep 07 16:36:11.164577 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:11.164585 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6474d96-9621-43cd-bfd9-98079628793b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-bd6c5a-c65f7fb55-smjqr\" (UID: \"e6474d96-9621-43cd-bfd9-98079628793b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bd6c5a-c65f7fb55-smjqr" Sep 07 16:36:11.165330 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:11.165307 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6474d96-9621-43cd-bfd9-98079628793b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-bd6c5a-c65f7fb55-smjqr\" (UID: \"e6474d96-9621-43cd-bfd9-98079628793b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bd6c5a-c65f7fb55-smjqr" Sep 07 16:36:11.173142 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:11.173121 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgbbd\" (UniqueName: \"kubernetes.io/projected/e6474d96-9621-43cd-bfd9-98079628793b-kube-api-access-fgbbd\") pod \"payload-processing-e2e-shared-a-w2-bd6c5a-c65f7fb55-smjqr\" (UID: \"e6474d96-9621-43cd-bfd9-98079628793b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bd6c5a-c65f7fb55-smjqr" Sep 07 16:36:11.273514 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:11.273419 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bd6c5a-c65f7fb55-smjqr" Sep 07 16:36:11.544647 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:11.544617 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-bd6c5a-c65f7fb55-smjqr"] Sep 07 16:36:11.547259 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:36:11.547237 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6474d96_9621_43cd_bfd9_98079628793b.slice/crio-124c1160ff7810f3ef416d7eff6c701e474d09359b0ff9c7ed4b2074789e0e59 WatchSource:0}: Error finding container 124c1160ff7810f3ef416d7eff6c701e474d09359b0ff9c7ed4b2074789e0e59: Status 404 returned error can't find the container with id 124c1160ff7810f3ef416d7eff6c701e474d09359b0ff9c7ed4b2074789e0e59 Sep 07 16:36:11.929370 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:11.929258 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bd6c5a-c65f7fb55-smjqr" event={"ID":"e6474d96-9621-43cd-bfd9-98079628793b","Type":"ContainerStarted","Data":"bd938ebb3aaaa7eacd5dee4a6eee812b3df07c1d91a708c3b8fec079c39c22b2"} Sep 07 16:36:11.929370 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:11.929325 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bd6c5a-c65f7fb55-smjqr" event={"ID":"e6474d96-9621-43cd-bfd9-98079628793b","Type":"ContainerStarted","Data":"124c1160ff7810f3ef416d7eff6c701e474d09359b0ff9c7ed4b2074789e0e59"} Sep 07 16:36:11.929370 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:11.929357 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bd6c5a-c65f7fb55-smjqr" Sep 07 16:36:11.947670 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:11.947604 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bd6c5a-c65f7fb55-smjqr" podStartSLOduration=1.9475808529999998 podStartE2EDuration="1.947580853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:36:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:36:11.946338057 +0000 UTC m=+1000.981561128" watchObservedRunningTime="2026-09-07 16:36:11.947580853 +0000 UTC m=+1000.982803931" Sep 07 16:36:12.825306 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:12.825271 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n"] Sep 07 16:36:12.828558 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:12.828535 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n" Sep 07 16:36:12.830751 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:12.830718 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-7aca0f-openshift-default-dockercfg-nksm6\"" Sep 07 16:36:12.841143 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:12.841114 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n"] Sep 07 16:36:12.880339 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:12.880299 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f620b12-a4d0-4b27-b702-be0f8c74e22e-credential-socket\") pod \"e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n\" (UID: \"7f620b12-a4d0-4b27-b702-be0f8c74e22e\") " pod="openshift-ingress/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n" Sep 07 16:36:12.880504 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:12.880350 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f620b12-a4d0-4b27-b702-be0f8c74e22e-istio-envoy\") pod \"e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n\" (UID: \"7f620b12-a4d0-4b27-b702-be0f8c74e22e\") " pod="openshift-ingress/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n" Sep 07 16:36:12.880504 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:12.880371 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26l88\" (UniqueName: \"kubernetes.io/projected/7f620b12-a4d0-4b27-b702-be0f8c74e22e-kube-api-access-26l88\") pod \"e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n\" (UID: \"7f620b12-a4d0-4b27-b702-be0f8c74e22e\") " pod="openshift-ingress/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n" Sep 07 16:36:12.880504 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:12.880455 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f620b12-a4d0-4b27-b702-be0f8c74e22e-workload-certs\") pod \"e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n\" (UID: \"7f620b12-a4d0-4b27-b702-be0f8c74e22e\") " pod="openshift-ingress/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n" Sep 07 16:36:12.880658 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:12.880508 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f620b12-a4d0-4b27-b702-be0f8c74e22e-istiod-ca-cert\") pod \"e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n\" (UID: \"7f620b12-a4d0-4b27-b702-be0f8c74e22e\") " pod="openshift-ingress/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n" Sep 07 16:36:12.880658 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:12.880559 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f620b12-a4d0-4b27-b702-be0f8c74e22e-istio-data\") pod \"e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n\" (UID: \"7f620b12-a4d0-4b27-b702-be0f8c74e22e\") " pod="openshift-ingress/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n" Sep 07 16:36:12.880658 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:12.880603 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f620b12-a4d0-4b27-b702-be0f8c74e22e-istio-token\") pod \"e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n\" (UID: \"7f620b12-a4d0-4b27-b702-be0f8c74e22e\") " pod="openshift-ingress/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n" Sep 07 16:36:12.880658 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:12.880639 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f620b12-a4d0-4b27-b702-be0f8c74e22e-istio-podinfo\") pod \"e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n\" (UID: \"7f620b12-a4d0-4b27-b702-be0f8c74e22e\") " pod="openshift-ingress/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n" Sep 07 16:36:12.880805 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:12.880671 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f620b12-a4d0-4b27-b702-be0f8c74e22e-workload-socket\") pod \"e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n\" (UID: \"7f620b12-a4d0-4b27-b702-be0f8c74e22e\") " pod="openshift-ingress/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n" Sep 07 16:36:12.981672 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:12.981635 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f620b12-a4d0-4b27-b702-be0f8c74e22e-istio-envoy\") pod \"e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n\" (UID: \"7f620b12-a4d0-4b27-b702-be0f8c74e22e\") " pod="openshift-ingress/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n" Sep 07 16:36:12.981672 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:12.981671 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26l88\" (UniqueName: \"kubernetes.io/projected/7f620b12-a4d0-4b27-b702-be0f8c74e22e-kube-api-access-26l88\") pod \"e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n\" (UID: \"7f620b12-a4d0-4b27-b702-be0f8c74e22e\") " pod="openshift-ingress/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n" Sep 07 16:36:12.981940 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:12.981693 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f620b12-a4d0-4b27-b702-be0f8c74e22e-workload-certs\") pod \"e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n\" (UID: \"7f620b12-a4d0-4b27-b702-be0f8c74e22e\") " pod="openshift-ingress/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n" Sep 07 16:36:12.981940 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:12.981717 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f620b12-a4d0-4b27-b702-be0f8c74e22e-istiod-ca-cert\") pod \"e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n\" (UID: \"7f620b12-a4d0-4b27-b702-be0f8c74e22e\") " pod="openshift-ingress/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n" Sep 07 16:36:12.981940 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:12.981737 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f620b12-a4d0-4b27-b702-be0f8c74e22e-istio-data\") pod \"e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n\" (UID: \"7f620b12-a4d0-4b27-b702-be0f8c74e22e\") " pod="openshift-ingress/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n" Sep 07 16:36:12.981940 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:12.981781 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f620b12-a4d0-4b27-b702-be0f8c74e22e-istio-token\") pod \"e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n\" (UID: \"7f620b12-a4d0-4b27-b702-be0f8c74e22e\") " pod="openshift-ingress/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n" Sep 07 16:36:12.981940 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:12.981834 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f620b12-a4d0-4b27-b702-be0f8c74e22e-istio-podinfo\") pod \"e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n\" (UID: \"7f620b12-a4d0-4b27-b702-be0f8c74e22e\") " pod="openshift-ingress/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n" Sep 07 16:36:12.981940 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:12.981869 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f620b12-a4d0-4b27-b702-be0f8c74e22e-workload-socket\") pod \"e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n\" (UID: \"7f620b12-a4d0-4b27-b702-be0f8c74e22e\") " pod="openshift-ingress/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n" Sep 07 16:36:12.981940 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:12.981924 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f620b12-a4d0-4b27-b702-be0f8c74e22e-credential-socket\") pod \"e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n\" (UID: \"7f620b12-a4d0-4b27-b702-be0f8c74e22e\") " pod="openshift-ingress/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n" Sep 07 16:36:12.982352 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:12.982194 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f620b12-a4d0-4b27-b702-be0f8c74e22e-workload-certs\") pod \"e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n\" (UID: \"7f620b12-a4d0-4b27-b702-be0f8c74e22e\") " pod="openshift-ingress/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n" Sep 07 16:36:12.982352 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:12.982242 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f620b12-a4d0-4b27-b702-be0f8c74e22e-istio-data\") pod \"e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n\" (UID: \"7f620b12-a4d0-4b27-b702-be0f8c74e22e\") " pod="openshift-ingress/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n" Sep 07 16:36:12.982623 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:12.982467 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f620b12-a4d0-4b27-b702-be0f8c74e22e-workload-socket\") pod \"e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n\" (UID: \"7f620b12-a4d0-4b27-b702-be0f8c74e22e\") " pod="openshift-ingress/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n" Sep 07 16:36:12.982623 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:12.982524 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f620b12-a4d0-4b27-b702-be0f8c74e22e-credential-socket\") pod \"e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n\" (UID: \"7f620b12-a4d0-4b27-b702-be0f8c74e22e\") " pod="openshift-ingress/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n" Sep 07 16:36:12.982623 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:12.982584 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f620b12-a4d0-4b27-b702-be0f8c74e22e-istiod-ca-cert\") pod \"e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n\" (UID: \"7f620b12-a4d0-4b27-b702-be0f8c74e22e\") " pod="openshift-ingress/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n" Sep 07 16:36:12.984419 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:12.984395 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f620b12-a4d0-4b27-b702-be0f8c74e22e-istio-envoy\") pod \"e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n\" (UID: \"7f620b12-a4d0-4b27-b702-be0f8c74e22e\") " pod="openshift-ingress/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n" Sep 07 16:36:12.984555 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:12.984533 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f620b12-a4d0-4b27-b702-be0f8c74e22e-istio-podinfo\") pod \"e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n\" (UID: \"7f620b12-a4d0-4b27-b702-be0f8c74e22e\") " pod="openshift-ingress/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n" Sep 07 16:36:12.988593 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:12.988563 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f620b12-a4d0-4b27-b702-be0f8c74e22e-istio-token\") pod \"e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n\" (UID: \"7f620b12-a4d0-4b27-b702-be0f8c74e22e\") " pod="openshift-ingress/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n" Sep 07 16:36:12.989105 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:12.989086 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26l88\" (UniqueName: \"kubernetes.io/projected/7f620b12-a4d0-4b27-b702-be0f8c74e22e-kube-api-access-26l88\") pod \"e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n\" (UID: \"7f620b12-a4d0-4b27-b702-be0f8c74e22e\") " pod="openshift-ingress/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n" Sep 07 16:36:13.142770 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:13.142670 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n" Sep 07 16:36:13.421617 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:13.421541 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n"] Sep 07 16:36:13.424579 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:36:13.424547 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f620b12_a4d0_4b27_b702_be0f8c74e22e.slice/crio-787879d7d5b343e40eb2d1c4e55518cfe508f091ec0e7ce740e671c79ce56266 WatchSource:0}: Error finding container 787879d7d5b343e40eb2d1c4e55518cfe508f091ec0e7ce740e671c79ce56266: Status 404 returned error can't find the container with id 787879d7d5b343e40eb2d1c4e55518cfe508f091ec0e7ce740e671c79ce56266 Sep 07 16:36:13.426948 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:13.426911 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:36:13.427015 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:13.426999 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:36:13.427079 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:13.427060 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:36:13.941091 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:13.941054 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n" event={"ID":"7f620b12-a4d0-4b27-b702-be0f8c74e22e","Type":"ContainerStarted","Data":"5df1cdc037c82bdb52ad2fbcdc47f42ac151332e9f5fd5ad0d680e864b15e87a"} Sep 07 16:36:13.941588 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:13.941117 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n" event={"ID":"7f620b12-a4d0-4b27-b702-be0f8c74e22e","Type":"ContainerStarted","Data":"787879d7d5b343e40eb2d1c4e55518cfe508f091ec0e7ce740e671c79ce56266"} Sep 07 16:36:13.966171 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:13.966090 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n" podStartSLOduration=1.96607189 podStartE2EDuration="1.96607189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:36:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:36:13.963985786 +0000 UTC m=+1002.999208869" watchObservedRunningTime="2026-09-07 16:36:13.96607189 +0000 UTC m=+1003.001294961" Sep 07 16:36:14.142911 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:14.142870 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n" Sep 07 16:36:14.148237 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:14.148212 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n" Sep 07 16:36:14.945432 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:14.945402 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n" Sep 07 16:36:14.946600 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:14.946583 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n" Sep 07 16:36:17.710855 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:17.710822 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7aca0f-79db6b5986-m7dgg"] Sep 07 16:36:17.714513 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:17.714490 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7aca0f-79db6b5986-m7dgg" Sep 07 16:36:17.716642 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:17.716613 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-7aca0f\"" Sep 07 16:36:17.716642 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:17.716637 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-7aca0f-dockercfg-gltjp\"" Sep 07 16:36:17.724680 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:17.724655 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7aca0f-79db6b5986-m7dgg"] Sep 07 16:36:17.828014 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:17.827976 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krkkc\" (UniqueName: \"kubernetes.io/projected/1ae56c63-f8be-4d84-ad66-8503d5ca3e0e-kube-api-access-krkkc\") pod \"payload-pre-processing-e2e-shared-b-w3-7aca0f-79db6b5986-m7dgg\" (UID: \"1ae56c63-f8be-4d84-ad66-8503d5ca3e0e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7aca0f-79db6b5986-m7dgg" Sep 07 16:36:17.828276 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:17.828080 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae56c63-f8be-4d84-ad66-8503d5ca3e0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-7aca0f-79db6b5986-m7dgg\" (UID: \"1ae56c63-f8be-4d84-ad66-8503d5ca3e0e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7aca0f-79db6b5986-m7dgg" Sep 07 16:36:17.929452 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:17.929407 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krkkc\" (UniqueName: \"kubernetes.io/projected/1ae56c63-f8be-4d84-ad66-8503d5ca3e0e-kube-api-access-krkkc\") pod \"payload-pre-processing-e2e-shared-b-w3-7aca0f-79db6b5986-m7dgg\" (UID: \"1ae56c63-f8be-4d84-ad66-8503d5ca3e0e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7aca0f-79db6b5986-m7dgg" Sep 07 16:36:17.929613 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:17.929468 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae56c63-f8be-4d84-ad66-8503d5ca3e0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-7aca0f-79db6b5986-m7dgg\" (UID: \"1ae56c63-f8be-4d84-ad66-8503d5ca3e0e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7aca0f-79db6b5986-m7dgg" Sep 07 16:36:17.930232 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:17.930211 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae56c63-f8be-4d84-ad66-8503d5ca3e0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-7aca0f-79db6b5986-m7dgg\" (UID: \"1ae56c63-f8be-4d84-ad66-8503d5ca3e0e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7aca0f-79db6b5986-m7dgg" Sep 07 16:36:17.937117 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:17.937099 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krkkc\" (UniqueName: \"kubernetes.io/projected/1ae56c63-f8be-4d84-ad66-8503d5ca3e0e-kube-api-access-krkkc\") pod \"payload-pre-processing-e2e-shared-b-w3-7aca0f-79db6b5986-m7dgg\" (UID: \"1ae56c63-f8be-4d84-ad66-8503d5ca3e0e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7aca0f-79db6b5986-m7dgg" Sep 07 16:36:18.026287 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:18.026198 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7aca0f-79db6b5986-m7dgg" Sep 07 16:36:18.300880 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:18.300854 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7aca0f-79db6b5986-m7dgg"] Sep 07 16:36:18.303403 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:36:18.303357 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ae56c63_f8be_4d84_ad66_8503d5ca3e0e.slice/crio-5e66558030f0d0e52ec299f06b5af0ba4534e405df74b699a65f9dc96bdacd42 WatchSource:0}: Error finding container 5e66558030f0d0e52ec299f06b5af0ba4534e405df74b699a65f9dc96bdacd42: Status 404 returned error can't find the container with id 5e66558030f0d0e52ec299f06b5af0ba4534e405df74b699a65f9dc96bdacd42 Sep 07 16:36:18.972528 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:18.972493 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7aca0f-79db6b5986-m7dgg" event={"ID":"1ae56c63-f8be-4d84-ad66-8503d5ca3e0e","Type":"ContainerStarted","Data":"f566236a5a8f88af44eb63952a0a7f05c71dc472f12eea54f08fbbda1269afb4"} Sep 07 16:36:18.972942 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:18.972544 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7aca0f-79db6b5986-m7dgg" event={"ID":"1ae56c63-f8be-4d84-ad66-8503d5ca3e0e","Type":"ContainerStarted","Data":"5e66558030f0d0e52ec299f06b5af0ba4534e405df74b699a65f9dc96bdacd42"} Sep 07 16:36:18.972942 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:18.972589 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7aca0f-79db6b5986-m7dgg" Sep 07 16:36:18.986737 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:18.986687 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7aca0f-79db6b5986-m7dgg" podStartSLOduration=1.986672098 podStartE2EDuration="1.986672098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:36:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:36:18.985943757 +0000 UTC m=+1008.021166829" watchObservedRunningTime="2026-09-07 16:36:18.986672098 +0000 UTC m=+1008.021895169" Sep 07 16:36:22.937311 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:22.937275 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bd6c5a-c65f7fb55-smjqr" Sep 07 16:36:29.979009 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:29.978973 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7aca0f-79db6b5986-m7dgg" Sep 07 16:36:30.844436 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:30.844221 2580 patch_prober.go:28] interesting pod/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": read tcp 10.133.0.2:53410->10.133.0.49:15021: read: connection reset by peer" start-of-body= Sep 07 16:36:30.844436 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:30.844337 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n" podUID="7f620b12-a4d0-4b27-b702-be0f8c74e22e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": read tcp 10.133.0.2:53410->10.133.0.49:15021: read: connection reset by peer" Sep 07 16:36:31.035645 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:31.035555 2580 generic.go:356] "Generic (PLEG): container finished" podID="7f620b12-a4d0-4b27-b702-be0f8c74e22e" containerID="5df1cdc037c82bdb52ad2fbcdc47f42ac151332e9f5fd5ad0d680e864b15e87a" exitCode=0 Sep 07 16:36:31.035645 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:31.035612 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n" event={"ID":"7f620b12-a4d0-4b27-b702-be0f8c74e22e","Type":"ContainerDied","Data":"5df1cdc037c82bdb52ad2fbcdc47f42ac151332e9f5fd5ad0d680e864b15e87a"} Sep 07 16:36:31.036059 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:31.035957 2580 scope.go:117] "RemoveContainer" containerID="5df1cdc037c82bdb52ad2fbcdc47f42ac151332e9f5fd5ad0d680e864b15e87a" Sep 07 16:36:31.037174 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:31.037145 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:36:31.037249 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:31.037231 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:36:31.037286 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:31.037276 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:36:32.042236 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:32.042197 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n" event={"ID":"7f620b12-a4d0-4b27-b702-be0f8c74e22e","Type":"ContainerStarted","Data":"1ff39bc4d94daf0681122a74ff8ce6844d4965eb2abe9fbb992b21206d2dab20"} Sep 07 16:36:32.143772 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:32.143723 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n" Sep 07 16:36:32.144009 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:32.143981 2580 patch_prober.go:28] interesting pod/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Sep 07 16:36:32.144153 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:32.144084 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n" podUID="7f620b12-a4d0-4b27-b702-be0f8c74e22e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Sep 07 16:36:33.143838 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:33.143795 2580 patch_prober.go:28] interesting pod/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Sep 07 16:36:33.144346 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:33.143866 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n" podUID="7f620b12-a4d0-4b27-b702-be0f8c74e22e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Sep 07 16:36:34.143696 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:34.143655 2580 patch_prober.go:28] interesting pod/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Sep 07 16:36:34.143917 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:34.143729 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n" podUID="7f620b12-a4d0-4b27-b702-be0f8c74e22e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Sep 07 16:36:35.143325 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:35.143288 2580 patch_prober.go:28] interesting pod/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Sep 07 16:36:35.143528 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:35.143349 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n" podUID="7f620b12-a4d0-4b27-b702-be0f8c74e22e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Sep 07 16:36:36.143318 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:36.143275 2580 patch_prober.go:28] interesting pod/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Sep 07 16:36:36.144238 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:36.144195 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n" podUID="7f620b12-a4d0-4b27-b702-be0f8c74e22e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Sep 07 16:36:37.143376 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:37.143330 2580 patch_prober.go:28] interesting pod/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Sep 07 16:36:37.143780 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:37.143397 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n" podUID="7f620b12-a4d0-4b27-b702-be0f8c74e22e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Sep 07 16:36:38.147565 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:38.147532 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n" Sep 07 16:36:38.148017 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:38.147824 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n" Sep 07 16:36:38.148519 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:38.148504 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n" Sep 07 16:36:38.544577 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:38.544545 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4a9a21-698c7fd6c9-4bdr6"] Sep 07 16:36:38.547880 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:38.547864 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4a9a21-698c7fd6c9-4bdr6" Sep 07 16:36:38.550065 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:38.550041 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-4a9a21\"" Sep 07 16:36:38.550212 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:38.550123 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-4a9a21-dockercfg-lrnnf\"" Sep 07 16:36:38.556930 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:38.556904 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4a9a21-698c7fd6c9-4bdr6"] Sep 07 16:36:38.622792 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:38.622757 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7j6k\" (UniqueName: \"kubernetes.io/projected/c90db097-6a82-43c5-bcb7-2f4603f76a5c-kube-api-access-x7j6k\") pod \"payload-processing-e2e-shared-b-w2-4a9a21-698c7fd6c9-4bdr6\" (UID: \"c90db097-6a82-43c5-bcb7-2f4603f76a5c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4a9a21-698c7fd6c9-4bdr6" Sep 07 16:36:38.623060 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:38.622868 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c90db097-6a82-43c5-bcb7-2f4603f76a5c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-4a9a21-698c7fd6c9-4bdr6\" (UID: \"c90db097-6a82-43c5-bcb7-2f4603f76a5c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4a9a21-698c7fd6c9-4bdr6" Sep 07 16:36:38.723524 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:38.723466 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7j6k\" (UniqueName: \"kubernetes.io/projected/c90db097-6a82-43c5-bcb7-2f4603f76a5c-kube-api-access-x7j6k\") pod \"payload-processing-e2e-shared-b-w2-4a9a21-698c7fd6c9-4bdr6\" (UID: \"c90db097-6a82-43c5-bcb7-2f4603f76a5c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4a9a21-698c7fd6c9-4bdr6" Sep 07 16:36:38.723710 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:38.723574 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c90db097-6a82-43c5-bcb7-2f4603f76a5c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-4a9a21-698c7fd6c9-4bdr6\" (UID: \"c90db097-6a82-43c5-bcb7-2f4603f76a5c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4a9a21-698c7fd6c9-4bdr6" Sep 07 16:36:38.724278 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:38.724260 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c90db097-6a82-43c5-bcb7-2f4603f76a5c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-4a9a21-698c7fd6c9-4bdr6\" (UID: \"c90db097-6a82-43c5-bcb7-2f4603f76a5c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4a9a21-698c7fd6c9-4bdr6" Sep 07 16:36:38.730547 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:38.730521 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7j6k\" (UniqueName: \"kubernetes.io/projected/c90db097-6a82-43c5-bcb7-2f4603f76a5c-kube-api-access-x7j6k\") pod \"payload-processing-e2e-shared-b-w2-4a9a21-698c7fd6c9-4bdr6\" (UID: \"c90db097-6a82-43c5-bcb7-2f4603f76a5c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4a9a21-698c7fd6c9-4bdr6" Sep 07 16:36:38.859265 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:38.859164 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4a9a21-698c7fd6c9-4bdr6" Sep 07 16:36:39.133352 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:39.133164 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4a9a21-698c7fd6c9-4bdr6"] Sep 07 16:36:39.135942 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:36:39.135905 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc90db097_6a82_43c5_bcb7_2f4603f76a5c.slice/crio-53e56e62908c54511c3e4a16fe6e4582a5c80fc840a50afa0f7ed9fe9e194e01 WatchSource:0}: Error finding container 53e56e62908c54511c3e4a16fe6e4582a5c80fc840a50afa0f7ed9fe9e194e01: Status 404 returned error can't find the container with id 53e56e62908c54511c3e4a16fe6e4582a5c80fc840a50afa0f7ed9fe9e194e01 Sep 07 16:36:40.083833 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:40.083801 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4a9a21-698c7fd6c9-4bdr6" event={"ID":"c90db097-6a82-43c5-bcb7-2f4603f76a5c","Type":"ContainerStarted","Data":"dc85920362207839b20d8d1d176c5068733bb618185bd408899d3ad04e013c46"} Sep 07 16:36:40.084319 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:40.083859 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4a9a21-698c7fd6c9-4bdr6" event={"ID":"c90db097-6a82-43c5-bcb7-2f4603f76a5c","Type":"ContainerStarted","Data":"53e56e62908c54511c3e4a16fe6e4582a5c80fc840a50afa0f7ed9fe9e194e01"} Sep 07 16:36:40.084319 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:40.083887 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4a9a21-698c7fd6c9-4bdr6" Sep 07 16:36:40.109675 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:40.109619 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4a9a21-698c7fd6c9-4bdr6" podStartSLOduration=2.109602767 podStartE2EDuration="2.109602767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:36:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:36:40.10791722 +0000 UTC m=+1029.143140303" watchObservedRunningTime="2026-09-07 16:36:40.109602767 +0000 UTC m=+1029.144825837" Sep 07 16:36:51.091057 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:36:51.090988 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4a9a21-698c7fd6c9-4bdr6" Sep 07 16:37:15.429942 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:15.429905 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8"] Sep 07 16:37:15.432503 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:15.432478 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8" Sep 07 16:37:15.435208 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:15.435185 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-2ff801c7-openshift-default-dockercfg-gjslj\"" Sep 07 16:37:15.446548 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:15.446515 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8"] Sep 07 16:37:15.532944 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:15.532896 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f27ee3f0-0ef7-4e40-904a-ca944d08f322-credential-socket\") pod \"e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8\" (UID: \"f27ee3f0-0ef7-4e40-904a-ca944d08f322\") " pod="openshift-ingress/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8" Sep 07 16:37:15.532944 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:15.532944 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f27ee3f0-0ef7-4e40-904a-ca944d08f322-istio-data\") pod \"e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8\" (UID: \"f27ee3f0-0ef7-4e40-904a-ca944d08f322\") " pod="openshift-ingress/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8" Sep 07 16:37:15.533220 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:15.532975 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f27ee3f0-0ef7-4e40-904a-ca944d08f322-istio-podinfo\") pod \"e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8\" (UID: \"f27ee3f0-0ef7-4e40-904a-ca944d08f322\") " pod="openshift-ingress/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8" Sep 07 16:37:15.533220 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:15.532998 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f27ee3f0-0ef7-4e40-904a-ca944d08f322-istiod-ca-cert\") pod \"e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8\" (UID: \"f27ee3f0-0ef7-4e40-904a-ca944d08f322\") " pod="openshift-ingress/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8" Sep 07 16:37:15.533220 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:15.533049 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn5fn\" (UniqueName: \"kubernetes.io/projected/f27ee3f0-0ef7-4e40-904a-ca944d08f322-kube-api-access-sn5fn\") pod \"e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8\" (UID: \"f27ee3f0-0ef7-4e40-904a-ca944d08f322\") " pod="openshift-ingress/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8" Sep 07 16:37:15.533220 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:15.533150 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f27ee3f0-0ef7-4e40-904a-ca944d08f322-istio-envoy\") pod \"e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8\" (UID: \"f27ee3f0-0ef7-4e40-904a-ca944d08f322\") " pod="openshift-ingress/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8" Sep 07 16:37:15.533436 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:15.533230 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f27ee3f0-0ef7-4e40-904a-ca944d08f322-workload-certs\") pod \"e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8\" (UID: \"f27ee3f0-0ef7-4e40-904a-ca944d08f322\") " pod="openshift-ingress/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8" Sep 07 16:37:15.533436 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:15.533262 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f27ee3f0-0ef7-4e40-904a-ca944d08f322-workload-socket\") pod \"e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8\" (UID: \"f27ee3f0-0ef7-4e40-904a-ca944d08f322\") " pod="openshift-ingress/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8" Sep 07 16:37:15.533436 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:15.533353 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f27ee3f0-0ef7-4e40-904a-ca944d08f322-istio-token\") pod \"e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8\" (UID: \"f27ee3f0-0ef7-4e40-904a-ca944d08f322\") " pod="openshift-ingress/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8" Sep 07 16:37:15.633780 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:15.633738 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f27ee3f0-0ef7-4e40-904a-ca944d08f322-istio-envoy\") pod \"e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8\" (UID: \"f27ee3f0-0ef7-4e40-904a-ca944d08f322\") " pod="openshift-ingress/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8" Sep 07 16:37:15.633955 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:15.633800 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f27ee3f0-0ef7-4e40-904a-ca944d08f322-workload-certs\") pod \"e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8\" (UID: \"f27ee3f0-0ef7-4e40-904a-ca944d08f322\") " pod="openshift-ingress/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8" Sep 07 16:37:15.633955 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:15.633828 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f27ee3f0-0ef7-4e40-904a-ca944d08f322-workload-socket\") pod \"e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8\" (UID: \"f27ee3f0-0ef7-4e40-904a-ca944d08f322\") " pod="openshift-ingress/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8" Sep 07 16:37:15.634087 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:15.633982 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f27ee3f0-0ef7-4e40-904a-ca944d08f322-istio-token\") pod \"e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8\" (UID: \"f27ee3f0-0ef7-4e40-904a-ca944d08f322\") " pod="openshift-ingress/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8" Sep 07 16:37:15.634150 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:15.634097 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f27ee3f0-0ef7-4e40-904a-ca944d08f322-credential-socket\") pod \"e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8\" (UID: \"f27ee3f0-0ef7-4e40-904a-ca944d08f322\") " pod="openshift-ingress/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8" Sep 07 16:37:15.634150 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:15.634125 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f27ee3f0-0ef7-4e40-904a-ca944d08f322-istio-data\") pod \"e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8\" (UID: \"f27ee3f0-0ef7-4e40-904a-ca944d08f322\") " pod="openshift-ingress/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8" Sep 07 16:37:15.634252 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:15.634158 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f27ee3f0-0ef7-4e40-904a-ca944d08f322-istio-podinfo\") pod \"e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8\" (UID: \"f27ee3f0-0ef7-4e40-904a-ca944d08f322\") " pod="openshift-ingress/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8" Sep 07 16:37:15.634252 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:15.634182 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f27ee3f0-0ef7-4e40-904a-ca944d08f322-istiod-ca-cert\") pod \"e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8\" (UID: \"f27ee3f0-0ef7-4e40-904a-ca944d08f322\") " pod="openshift-ingress/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8" Sep 07 16:37:15.634252 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:15.634211 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn5fn\" (UniqueName: \"kubernetes.io/projected/f27ee3f0-0ef7-4e40-904a-ca944d08f322-kube-api-access-sn5fn\") pod \"e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8\" (UID: \"f27ee3f0-0ef7-4e40-904a-ca944d08f322\") " pod="openshift-ingress/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8" Sep 07 16:37:15.634424 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:15.634268 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f27ee3f0-0ef7-4e40-904a-ca944d08f322-workload-certs\") pod \"e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8\" (UID: \"f27ee3f0-0ef7-4e40-904a-ca944d08f322\") " pod="openshift-ingress/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8" Sep 07 16:37:15.634539 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:15.634513 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f27ee3f0-0ef7-4e40-904a-ca944d08f322-credential-socket\") pod \"e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8\" (UID: \"f27ee3f0-0ef7-4e40-904a-ca944d08f322\") " pod="openshift-ingress/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8" Sep 07 16:37:15.634604 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:15.634538 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f27ee3f0-0ef7-4e40-904a-ca944d08f322-istio-data\") pod \"e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8\" (UID: \"f27ee3f0-0ef7-4e40-904a-ca944d08f322\") " pod="openshift-ingress/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8" Sep 07 16:37:15.634653 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:15.634637 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f27ee3f0-0ef7-4e40-904a-ca944d08f322-workload-socket\") pod \"e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8\" (UID: \"f27ee3f0-0ef7-4e40-904a-ca944d08f322\") " pod="openshift-ingress/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8" Sep 07 16:37:15.635153 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:15.635059 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f27ee3f0-0ef7-4e40-904a-ca944d08f322-istiod-ca-cert\") pod \"e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8\" (UID: \"f27ee3f0-0ef7-4e40-904a-ca944d08f322\") " pod="openshift-ingress/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8" Sep 07 16:37:15.636359 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:15.636334 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f27ee3f0-0ef7-4e40-904a-ca944d08f322-istio-envoy\") pod \"e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8\" (UID: \"f27ee3f0-0ef7-4e40-904a-ca944d08f322\") " pod="openshift-ingress/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8" Sep 07 16:37:15.636851 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:15.636832 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f27ee3f0-0ef7-4e40-904a-ca944d08f322-istio-podinfo\") pod \"e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8\" (UID: \"f27ee3f0-0ef7-4e40-904a-ca944d08f322\") " pod="openshift-ingress/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8" Sep 07 16:37:15.641241 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:15.641214 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f27ee3f0-0ef7-4e40-904a-ca944d08f322-istio-token\") pod \"e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8\" (UID: \"f27ee3f0-0ef7-4e40-904a-ca944d08f322\") " pod="openshift-ingress/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8" Sep 07 16:37:15.641592 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:15.641573 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn5fn\" (UniqueName: \"kubernetes.io/projected/f27ee3f0-0ef7-4e40-904a-ca944d08f322-kube-api-access-sn5fn\") pod \"e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8\" (UID: \"f27ee3f0-0ef7-4e40-904a-ca944d08f322\") " pod="openshift-ingress/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8" Sep 07 16:37:15.743258 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:15.743225 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8" Sep 07 16:37:16.018786 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:16.018747 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8"] Sep 07 16:37:16.021310 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:37:16.021275 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf27ee3f0_0ef7_4e40_904a_ca944d08f322.slice/crio-50d5a106fd89d0782d3755cc3236f3ad95931a895943f39af54b6d8c9a83e1f0 WatchSource:0}: Error finding container 50d5a106fd89d0782d3755cc3236f3ad95931a895943f39af54b6d8c9a83e1f0: Status 404 returned error can't find the container with id 50d5a106fd89d0782d3755cc3236f3ad95931a895943f39af54b6d8c9a83e1f0 Sep 07 16:37:16.023909 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:16.023877 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:37:16.024042 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:16.023954 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:37:16.024042 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:16.023999 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:37:16.268646 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:16.268556 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8" event={"ID":"f27ee3f0-0ef7-4e40-904a-ca944d08f322","Type":"ContainerStarted","Data":"1d957e8baa80e62a8246fe26bb37ff8a70fe44483c606fd14fefecad1c40177a"} Sep 07 16:37:16.268646 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:16.268608 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8" event={"ID":"f27ee3f0-0ef7-4e40-904a-ca944d08f322","Type":"ContainerStarted","Data":"50d5a106fd89d0782d3755cc3236f3ad95931a895943f39af54b6d8c9a83e1f0"} Sep 07 16:37:16.287135 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:16.287068 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8" podStartSLOduration=1.287045955 podStartE2EDuration="1.287045955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:37:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:37:16.284755922 +0000 UTC m=+1065.319979007" watchObservedRunningTime="2026-09-07 16:37:16.287045955 +0000 UTC m=+1065.322269027" Sep 07 16:37:16.743364 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:16.743325 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8" Sep 07 16:37:17.749742 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:17.749707 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8" Sep 07 16:37:18.277732 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:18.277699 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8" Sep 07 16:37:18.278847 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:18.278821 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8" Sep 07 16:37:19.938748 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:19.938708 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2ff801c7-7d46558dfd-f22x8"] Sep 07 16:37:19.941353 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:19.941330 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2ff801c7-7d46558dfd-f22x8" Sep 07 16:37:19.943718 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:19.943695 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-2ff801c7\"" Sep 07 16:37:19.943718 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:19.943710 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-2ff801c7-dockercfg-nm887\"" Sep 07 16:37:19.951584 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:19.951553 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2ff801c7-7d46558dfd-f22x8"] Sep 07 16:37:19.977327 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:19.977292 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2ff801c7-58777bb965-2sxcz"] Sep 07 16:37:19.977998 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:19.977976 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z87st\" (UniqueName: \"kubernetes.io/projected/3a98227c-12fd-4da2-b170-2f682d0c4818-kube-api-access-z87st\") pod \"payload-pre-processing-e2e-mt-2ff801c7-7d46558dfd-f22x8\" (UID: \"3a98227c-12fd-4da2-b170-2f682d0c4818\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2ff801c7-7d46558dfd-f22x8" Sep 07 16:37:19.978145 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:19.978130 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a98227c-12fd-4da2-b170-2f682d0c4818-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2ff801c7-7d46558dfd-f22x8\" (UID: \"3a98227c-12fd-4da2-b170-2f682d0c4818\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2ff801c7-7d46558dfd-f22x8" Sep 07 16:37:19.979622 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:19.979608 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2ff801c7-58777bb965-2sxcz" Sep 07 16:37:19.990598 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:19.990575 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2ff801c7-58777bb965-2sxcz"] Sep 07 16:37:20.079355 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:20.079316 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12647717-ec9c-4960-ac57-a0ee0aa001b6-plugins-config-volume\") pod \"payload-processing-e2e-mt-2ff801c7-58777bb965-2sxcz\" (UID: \"12647717-ec9c-4960-ac57-a0ee0aa001b6\") " pod="openshift-ingress/payload-processing-e2e-mt-2ff801c7-58777bb965-2sxcz" Sep 07 16:37:20.079549 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:20.079393 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a98227c-12fd-4da2-b170-2f682d0c4818-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2ff801c7-7d46558dfd-f22x8\" (UID: \"3a98227c-12fd-4da2-b170-2f682d0c4818\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2ff801c7-7d46558dfd-f22x8" Sep 07 16:37:20.079549 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:20.079431 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z87st\" (UniqueName: \"kubernetes.io/projected/3a98227c-12fd-4da2-b170-2f682d0c4818-kube-api-access-z87st\") pod \"payload-pre-processing-e2e-mt-2ff801c7-7d46558dfd-f22x8\" (UID: \"3a98227c-12fd-4da2-b170-2f682d0c4818\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2ff801c7-7d46558dfd-f22x8" Sep 07 16:37:20.079549 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:20.079475 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsq7f\" (UniqueName: \"kubernetes.io/projected/12647717-ec9c-4960-ac57-a0ee0aa001b6-kube-api-access-jsq7f\") pod \"payload-processing-e2e-mt-2ff801c7-58777bb965-2sxcz\" (UID: \"12647717-ec9c-4960-ac57-a0ee0aa001b6\") " pod="openshift-ingress/payload-processing-e2e-mt-2ff801c7-58777bb965-2sxcz" Sep 07 16:37:20.080144 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:20.080125 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a98227c-12fd-4da2-b170-2f682d0c4818-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2ff801c7-7d46558dfd-f22x8\" (UID: \"3a98227c-12fd-4da2-b170-2f682d0c4818\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2ff801c7-7d46558dfd-f22x8" Sep 07 16:37:20.087091 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:20.087070 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z87st\" (UniqueName: \"kubernetes.io/projected/3a98227c-12fd-4da2-b170-2f682d0c4818-kube-api-access-z87st\") pod \"payload-pre-processing-e2e-mt-2ff801c7-7d46558dfd-f22x8\" (UID: \"3a98227c-12fd-4da2-b170-2f682d0c4818\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2ff801c7-7d46558dfd-f22x8" Sep 07 16:37:20.180879 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:20.180841 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsq7f\" (UniqueName: \"kubernetes.io/projected/12647717-ec9c-4960-ac57-a0ee0aa001b6-kube-api-access-jsq7f\") pod \"payload-processing-e2e-mt-2ff801c7-58777bb965-2sxcz\" (UID: \"12647717-ec9c-4960-ac57-a0ee0aa001b6\") " pod="openshift-ingress/payload-processing-e2e-mt-2ff801c7-58777bb965-2sxcz" Sep 07 16:37:20.181078 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:20.180908 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12647717-ec9c-4960-ac57-a0ee0aa001b6-plugins-config-volume\") pod \"payload-processing-e2e-mt-2ff801c7-58777bb965-2sxcz\" (UID: \"12647717-ec9c-4960-ac57-a0ee0aa001b6\") " pod="openshift-ingress/payload-processing-e2e-mt-2ff801c7-58777bb965-2sxcz" Sep 07 16:37:20.181558 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:20.181539 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12647717-ec9c-4960-ac57-a0ee0aa001b6-plugins-config-volume\") pod \"payload-processing-e2e-mt-2ff801c7-58777bb965-2sxcz\" (UID: \"12647717-ec9c-4960-ac57-a0ee0aa001b6\") " pod="openshift-ingress/payload-processing-e2e-mt-2ff801c7-58777bb965-2sxcz" Sep 07 16:37:20.188457 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:20.188433 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsq7f\" (UniqueName: \"kubernetes.io/projected/12647717-ec9c-4960-ac57-a0ee0aa001b6-kube-api-access-jsq7f\") pod \"payload-processing-e2e-mt-2ff801c7-58777bb965-2sxcz\" (UID: \"12647717-ec9c-4960-ac57-a0ee0aa001b6\") " pod="openshift-ingress/payload-processing-e2e-mt-2ff801c7-58777bb965-2sxcz" Sep 07 16:37:20.254416 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:20.254361 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2ff801c7-7d46558dfd-f22x8" Sep 07 16:37:20.289942 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:20.289910 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2ff801c7-58777bb965-2sxcz" Sep 07 16:37:20.526687 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:20.526651 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2ff801c7-7d46558dfd-f22x8"] Sep 07 16:37:20.528860 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:37:20.528824 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a98227c_12fd_4da2_b170_2f682d0c4818.slice/crio-92046d75a416b8372b45afd64f5b51d4514df4f56a9b5d61a1bd4f0e21af12de WatchSource:0}: Error finding container 92046d75a416b8372b45afd64f5b51d4514df4f56a9b5d61a1bd4f0e21af12de: Status 404 returned error can't find the container with id 92046d75a416b8372b45afd64f5b51d4514df4f56a9b5d61a1bd4f0e21af12de Sep 07 16:37:20.568695 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:20.568670 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2ff801c7-58777bb965-2sxcz"] Sep 07 16:37:20.580230 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:37:20.580202 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12647717_ec9c_4960_ac57_a0ee0aa001b6.slice/crio-ea85ced71fef6cf6c1b53fd5fef36b51f27a75b8fdf9c55cb2b02eb3341a06a6 WatchSource:0}: Error finding container ea85ced71fef6cf6c1b53fd5fef36b51f27a75b8fdf9c55cb2b02eb3341a06a6: Status 404 returned error can't find the container with id ea85ced71fef6cf6c1b53fd5fef36b51f27a75b8fdf9c55cb2b02eb3341a06a6 Sep 07 16:37:21.294887 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:21.294847 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2ff801c7-7d46558dfd-f22x8" event={"ID":"3a98227c-12fd-4da2-b170-2f682d0c4818","Type":"ContainerStarted","Data":"7f4882215a5b7b24b4c56b94a74e8413b4e8cb457c266d188f13107000eea6f3"} Sep 07 16:37:21.295414 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:21.294906 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2ff801c7-7d46558dfd-f22x8" event={"ID":"3a98227c-12fd-4da2-b170-2f682d0c4818","Type":"ContainerStarted","Data":"92046d75a416b8372b45afd64f5b51d4514df4f56a9b5d61a1bd4f0e21af12de"} Sep 07 16:37:21.295414 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:21.294935 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-2ff801c7-7d46558dfd-f22x8" Sep 07 16:37:21.296559 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:21.296533 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2ff801c7-58777bb965-2sxcz" event={"ID":"12647717-ec9c-4960-ac57-a0ee0aa001b6","Type":"ContainerStarted","Data":"6067933ad1b78f4e08c9f0b3fa19c1e2f438c341bf483e8a7fac2ac0012864e9"} Sep 07 16:37:21.296664 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:21.296578 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2ff801c7-58777bb965-2sxcz" event={"ID":"12647717-ec9c-4960-ac57-a0ee0aa001b6","Type":"ContainerStarted","Data":"ea85ced71fef6cf6c1b53fd5fef36b51f27a75b8fdf9c55cb2b02eb3341a06a6"} Sep 07 16:37:21.296664 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:21.296599 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-2ff801c7-58777bb965-2sxcz" Sep 07 16:37:21.311730 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:21.311675 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-2ff801c7-7d46558dfd-f22x8" podStartSLOduration=2.311660468 podStartE2EDuration="2.311660468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:37:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:37:21.308701862 +0000 UTC m=+1070.343924947" watchObservedRunningTime="2026-09-07 16:37:21.311660468 +0000 UTC m=+1070.346883598" Sep 07 16:37:21.324494 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:21.324433 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-2ff801c7-58777bb965-2sxcz" podStartSLOduration=2.324417114 podStartE2EDuration="2.324417114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:37:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:37:21.322120372 +0000 UTC m=+1070.357343443" watchObservedRunningTime="2026-09-07 16:37:21.324417114 +0000 UTC m=+1070.359640183" Sep 07 16:37:31.518762 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:31.518653 2580 scope.go:117] "RemoveContainer" containerID="d31551bdd53f454cc6a17cebe9bbfef8611a149de37a9a7dab2dabc26fb2709e" Sep 07 16:37:31.535561 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:31.535530 2580 scope.go:117] "RemoveContainer" containerID="34f06df237fd476ee6c93a5d35feb5c0ca0d03006cf80c003e5065ecba2091c9" Sep 07 16:37:32.304382 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:32.304352 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-2ff801c7-58777bb965-2sxcz" Sep 07 16:37:32.304603 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:32.304420 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-2ff801c7-7d46558dfd-f22x8" Sep 07 16:37:34.319276 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:34.319221 2580 patch_prober.go:28] interesting pod/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 07 16:37:34.319681 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:34.319315 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8" podUID="f27ee3f0-0ef7-4e40-904a-ca944d08f322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 07 16:37:35.376517 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:35.376484 2580 generic.go:356] "Generic (PLEG): container finished" podID="f27ee3f0-0ef7-4e40-904a-ca944d08f322" containerID="1d957e8baa80e62a8246fe26bb37ff8a70fe44483c606fd14fefecad1c40177a" exitCode=0 Sep 07 16:37:35.376946 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:35.376559 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8" event={"ID":"f27ee3f0-0ef7-4e40-904a-ca944d08f322","Type":"ContainerDied","Data":"1d957e8baa80e62a8246fe26bb37ff8a70fe44483c606fd14fefecad1c40177a"} Sep 07 16:37:35.377080 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:35.377062 2580 scope.go:117] "RemoveContainer" containerID="1d957e8baa80e62a8246fe26bb37ff8a70fe44483c606fd14fefecad1c40177a" Sep 07 16:37:35.378244 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:35.378214 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:37:35.378302 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:35.378286 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:37:35.378348 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:35.378325 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:37:36.383821 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:36.383782 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8" event={"ID":"f27ee3f0-0ef7-4e40-904a-ca944d08f322","Type":"ContainerStarted","Data":"91cccbe5b6301b69db032e6a666ee4782729073e6c750e6fa951256c9fb9fee2"} Sep 07 16:37:36.743572 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:36.743523 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8" Sep 07 16:37:36.743869 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:36.743841 2580 patch_prober.go:28] interesting pod/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Sep 07 16:37:36.743940 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:36.743917 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8" podUID="f27ee3f0-0ef7-4e40-904a-ca944d08f322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Sep 07 16:37:37.744142 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:37.744098 2580 patch_prober.go:28] interesting pod/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Sep 07 16:37:37.744566 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:37.744162 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8" podUID="f27ee3f0-0ef7-4e40-904a-ca944d08f322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Sep 07 16:37:38.743974 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:38.743928 2580 patch_prober.go:28] interesting pod/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Sep 07 16:37:38.744206 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:38.744002 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8" podUID="f27ee3f0-0ef7-4e40-904a-ca944d08f322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Sep 07 16:37:39.744447 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:39.744401 2580 patch_prober.go:28] interesting pod/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Sep 07 16:37:39.744864 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:39.744470 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8" podUID="f27ee3f0-0ef7-4e40-904a-ca944d08f322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Sep 07 16:37:40.744073 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:40.744009 2580 patch_prober.go:28] interesting pod/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Sep 07 16:37:40.744260 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:40.744095 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8" podUID="f27ee3f0-0ef7-4e40-904a-ca944d08f322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Sep 07 16:37:41.821652 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:41.821620 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8" Sep 07 16:37:41.822102 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:41.822006 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8" Sep 07 16:37:41.822759 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:37:41.822740 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8" Sep 07 16:38:16.919507 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:38:16.919454 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2ff801c7-7kj5c"] Sep 07 16:38:16.922886 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:38:16.922869 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2ff801c7-7kj5c" Sep 07 16:38:16.925265 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:38:16.925246 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 07 16:38:16.925372 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:38:16.925338 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jpjxt\"" Sep 07 16:38:16.933850 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:38:16.933820 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2ff801c7-7kj5c"] Sep 07 16:38:17.023899 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:38:17.023855 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/786fae7b-5296-4bf2-9270-312191ed73b2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2ff801c7-7kj5c\" (UID: \"786fae7b-5296-4bf2-9270-312191ed73b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2ff801c7-7kj5c" Sep 07 16:38:17.125389 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:38:17.125352 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/786fae7b-5296-4bf2-9270-312191ed73b2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2ff801c7-7kj5c\" (UID: \"786fae7b-5296-4bf2-9270-312191ed73b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2ff801c7-7kj5c" Sep 07 16:38:17.125963 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:38:17.125942 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/786fae7b-5296-4bf2-9270-312191ed73b2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2ff801c7-7kj5c\" (UID: \"786fae7b-5296-4bf2-9270-312191ed73b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2ff801c7-7kj5c" Sep 07 16:38:17.233356 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:38:17.233320 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2ff801c7-7kj5c" Sep 07 16:38:17.508302 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:38:17.508233 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2ff801c7-7kj5c"] Sep 07 16:38:17.511037 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:38:17.510989 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod786fae7b_5296_4bf2_9270_312191ed73b2.slice/crio-08fd2ebb3d9adb15b3af71f02c037993ed70d9a9a48994ae644c30eb37dd8224 WatchSource:0}: Error finding container 08fd2ebb3d9adb15b3af71f02c037993ed70d9a9a48994ae644c30eb37dd8224: Status 404 returned error can't find the container with id 08fd2ebb3d9adb15b3af71f02c037993ed70d9a9a48994ae644c30eb37dd8224 Sep 07 16:38:17.588263 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:38:17.588230 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2ff801c7-7kj5c" event={"ID":"786fae7b-5296-4bf2-9270-312191ed73b2","Type":"ContainerStarted","Data":"08fd2ebb3d9adb15b3af71f02c037993ed70d9a9a48994ae644c30eb37dd8224"} Sep 07 16:38:18.596668 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:38:18.596627 2580 generic.go:356] "Generic (PLEG): container finished" podID="786fae7b-5296-4bf2-9270-312191ed73b2" containerID="7d0e15fc995b4c368b5ef7b0a8d9bcb8c5a6afe5593a255af29d76d73fbf841c" exitCode=0 Sep 07 16:38:18.597060 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:38:18.596683 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2ff801c7-7kj5c" event={"ID":"786fae7b-5296-4bf2-9270-312191ed73b2","Type":"ContainerDied","Data":"7d0e15fc995b4c368b5ef7b0a8d9bcb8c5a6afe5593a255af29d76d73fbf841c"} Sep 07 16:38:19.731607 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:38:19.731582 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2ff801c7-7kj5c" Sep 07 16:38:19.852651 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:38:19.852567 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/786fae7b-5296-4bf2-9270-312191ed73b2-maas-api-service-ca\") pod \"786fae7b-5296-4bf2-9270-312191ed73b2\" (UID: \"786fae7b-5296-4bf2-9270-312191ed73b2\") " Sep 07 16:38:19.852899 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:38:19.852876 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/786fae7b-5296-4bf2-9270-312191ed73b2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "786fae7b-5296-4bf2-9270-312191ed73b2" (UID: "786fae7b-5296-4bf2-9270-312191ed73b2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 16:38:19.953638 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:38:19.953597 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/786fae7b-5296-4bf2-9270-312191ed73b2-maas-api-service-ca\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:38:20.608341 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:38:20.608314 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2ff801c7-7kj5c" Sep 07 16:38:20.608545 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:38:20.608350 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2ff801c7-7kj5c" event={"ID":"786fae7b-5296-4bf2-9270-312191ed73b2","Type":"ContainerDied","Data":"08fd2ebb3d9adb15b3af71f02c037993ed70d9a9a48994ae644c30eb37dd8224"} Sep 07 16:38:20.608545 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:38:20.608401 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08fd2ebb3d9adb15b3af71f02c037993ed70d9a9a48994ae644c30eb37dd8224" Sep 07 16:38:31.558861 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:38:31.558830 2580 scope.go:117] "RemoveContainer" containerID="bd0802133c9fe0d9c9c58a111e03b2536999215571520ee9a32872fd6936aa39" Sep 07 16:38:32.528510 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:38:32.528470 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2ff801c7-58777bb965-2sxcz"] Sep 07 16:38:32.528805 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:38:32.528745 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-2ff801c7-58777bb965-2sxcz" podUID="12647717-ec9c-4960-ac57-a0ee0aa001b6" containerName="payload-processing" containerID="cri-o://6067933ad1b78f4e08c9f0b3fa19c1e2f438c341bf483e8a7fac2ac0012864e9" gracePeriod=30 Sep 07 16:38:32.535058 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:38:32.535006 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2ff801c7-7d46558dfd-f22x8"] Sep 07 16:38:32.535814 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:38:32.535785 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-2ff801c7-7d46558dfd-f22x8" podUID="3a98227c-12fd-4da2-b170-2f682d0c4818" containerName="payload-pre-processing" containerID="cri-o://7f4882215a5b7b24b4c56b94a74e8413b4e8cb457c266d188f13107000eea6f3" gracePeriod=30 Sep 07 16:38:32.668335 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:38:32.668300 2580 generic.go:356] "Generic (PLEG): container finished" podID="3a98227c-12fd-4da2-b170-2f682d0c4818" containerID="7f4882215a5b7b24b4c56b94a74e8413b4e8cb457c266d188f13107000eea6f3" exitCode=0 Sep 07 16:38:32.668719 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:38:32.668376 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2ff801c7-7d46558dfd-f22x8" event={"ID":"3a98227c-12fd-4da2-b170-2f682d0c4818","Type":"ContainerDied","Data":"7f4882215a5b7b24b4c56b94a74e8413b4e8cb457c266d188f13107000eea6f3"} Sep 07 16:38:32.670259 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:38:32.670233 2580 generic.go:356] "Generic (PLEG): container finished" podID="12647717-ec9c-4960-ac57-a0ee0aa001b6" containerID="6067933ad1b78f4e08c9f0b3fa19c1e2f438c341bf483e8a7fac2ac0012864e9" exitCode=0 Sep 07 16:38:32.670396 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:38:32.670267 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2ff801c7-58777bb965-2sxcz" event={"ID":"12647717-ec9c-4960-ac57-a0ee0aa001b6","Type":"ContainerDied","Data":"6067933ad1b78f4e08c9f0b3fa19c1e2f438c341bf483e8a7fac2ac0012864e9"} Sep 07 16:38:32.676625 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:38:32.676589 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2ff801c7-7kj5c"] Sep 07 16:38:32.679819 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:38:32.679794 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2ff801c7-7kj5c"] Sep 07 16:38:32.829169 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:38:32.829140 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2ff801c7-7d46558dfd-f22x8" Sep 07 16:38:32.831737 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:38:32.831716 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2ff801c7-58777bb965-2sxcz" Sep 07 16:38:32.981398 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:38:32.981358 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a98227c-12fd-4da2-b170-2f682d0c4818-plugins-config-volume\") pod \"3a98227c-12fd-4da2-b170-2f682d0c4818\" (UID: \"3a98227c-12fd-4da2-b170-2f682d0c4818\") " Sep 07 16:38:32.981578 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:38:32.981433 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z87st\" (UniqueName: \"kubernetes.io/projected/3a98227c-12fd-4da2-b170-2f682d0c4818-kube-api-access-z87st\") pod \"3a98227c-12fd-4da2-b170-2f682d0c4818\" (UID: \"3a98227c-12fd-4da2-b170-2f682d0c4818\") " Sep 07 16:38:32.981578 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:38:32.981451 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12647717-ec9c-4960-ac57-a0ee0aa001b6-plugins-config-volume\") pod \"12647717-ec9c-4960-ac57-a0ee0aa001b6\" (UID: \"12647717-ec9c-4960-ac57-a0ee0aa001b6\") " Sep 07 16:38:32.981578 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:38:32.981477 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jsq7f\" (UniqueName: \"kubernetes.io/projected/12647717-ec9c-4960-ac57-a0ee0aa001b6-kube-api-access-jsq7f\") pod \"12647717-ec9c-4960-ac57-a0ee0aa001b6\" (UID: \"12647717-ec9c-4960-ac57-a0ee0aa001b6\") " Sep 07 16:38:32.981807 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:38:32.981782 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12647717-ec9c-4960-ac57-a0ee0aa001b6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "12647717-ec9c-4960-ac57-a0ee0aa001b6" (UID: "12647717-ec9c-4960-ac57-a0ee0aa001b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 16:38:32.981859 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:38:32.981816 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a98227c-12fd-4da2-b170-2f682d0c4818-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3a98227c-12fd-4da2-b170-2f682d0c4818" (UID: "3a98227c-12fd-4da2-b170-2f682d0c4818"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 16:38:32.983881 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:38:32.983849 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12647717-ec9c-4960-ac57-a0ee0aa001b6-kube-api-access-jsq7f" (OuterVolumeSpecName: "kube-api-access-jsq7f") pod "12647717-ec9c-4960-ac57-a0ee0aa001b6" (UID: "12647717-ec9c-4960-ac57-a0ee0aa001b6"). InnerVolumeSpecName "kube-api-access-jsq7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 16:38:32.983980 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:38:32.983908 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a98227c-12fd-4da2-b170-2f682d0c4818-kube-api-access-z87st" (OuterVolumeSpecName: "kube-api-access-z87st") pod "3a98227c-12fd-4da2-b170-2f682d0c4818" (UID: "3a98227c-12fd-4da2-b170-2f682d0c4818"). InnerVolumeSpecName "kube-api-access-z87st". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 16:38:33.082879 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:38:33.082788 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a98227c-12fd-4da2-b170-2f682d0c4818-plugins-config-volume\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:38:33.082879 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:38:33.082820 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z87st\" (UniqueName: \"kubernetes.io/projected/3a98227c-12fd-4da2-b170-2f682d0c4818-kube-api-access-z87st\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:38:33.082879 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:38:33.082831 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12647717-ec9c-4960-ac57-a0ee0aa001b6-plugins-config-volume\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:38:33.082879 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:38:33.082840 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsq7f\" (UniqueName: \"kubernetes.io/projected/12647717-ec9c-4960-ac57-a0ee0aa001b6-kube-api-access-jsq7f\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:38:33.419248 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:38:33.419168 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="786fae7b-5296-4bf2-9270-312191ed73b2" path="/var/lib/kubelet/pods/786fae7b-5296-4bf2-9270-312191ed73b2/volumes" Sep 07 16:38:33.676876 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:38:33.676790 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2ff801c7-7d46558dfd-f22x8" Sep 07 16:38:33.676876 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:38:33.676805 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2ff801c7-7d46558dfd-f22x8" event={"ID":"3a98227c-12fd-4da2-b170-2f682d0c4818","Type":"ContainerDied","Data":"92046d75a416b8372b45afd64f5b51d4514df4f56a9b5d61a1bd4f0e21af12de"} Sep 07 16:38:33.676876 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:38:33.676875 2580 scope.go:117] "RemoveContainer" containerID="7f4882215a5b7b24b4c56b94a74e8413b4e8cb457c266d188f13107000eea6f3" Sep 07 16:38:33.678218 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:38:33.678204 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2ff801c7-58777bb965-2sxcz" Sep 07 16:38:33.678284 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:38:33.678238 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2ff801c7-58777bb965-2sxcz" event={"ID":"12647717-ec9c-4960-ac57-a0ee0aa001b6","Type":"ContainerDied","Data":"ea85ced71fef6cf6c1b53fd5fef36b51f27a75b8fdf9c55cb2b02eb3341a06a6"} Sep 07 16:38:33.685788 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:38:33.685770 2580 scope.go:117] "RemoveContainer" containerID="6067933ad1b78f4e08c9f0b3fa19c1e2f438c341bf483e8a7fac2ac0012864e9" Sep 07 16:38:33.693178 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:38:33.693145 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2ff801c7-7d46558dfd-f22x8"] Sep 07 16:38:33.696917 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:38:33.696886 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2ff801c7-7d46558dfd-f22x8"] Sep 07 16:38:33.706503 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:38:33.706477 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2ff801c7-58777bb965-2sxcz"] Sep 07 16:38:33.710775 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:38:33.710747 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2ff801c7-58777bb965-2sxcz"] Sep 07 16:38:35.418078 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:38:35.418012 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12647717-ec9c-4960-ac57-a0ee0aa001b6" path="/var/lib/kubelet/pods/12647717-ec9c-4960-ac57-a0ee0aa001b6/volumes" Sep 07 16:38:35.418510 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:38:35.418494 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a98227c-12fd-4da2-b170-2f682d0c4818" path="/var/lib/kubelet/pods/3a98227c-12fd-4da2-b170-2f682d0c4818/volumes" Sep 07 16:39:23.611602 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:39:23.611558 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8"] Sep 07 16:39:23.612150 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:39:23.611856 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8" podUID="f27ee3f0-0ef7-4e40-904a-ca944d08f322" containerName="istio-proxy" containerID="cri-o://91cccbe5b6301b69db032e6a666ee4782729073e6c750e6fa951256c9fb9fee2" gracePeriod=30 Sep 07 16:39:26.824336 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:39:26.824293 2580 patch_prober.go:28] interesting pod/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 07 16:39:26.824734 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:39:26.824349 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8" podUID="f27ee3f0-0ef7-4e40-904a-ca944d08f322" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 07 16:39:28.864445 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:39:28.864420 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8" Sep 07 16:39:28.893431 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:39:28.893319 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sn5fn\" (UniqueName: \"kubernetes.io/projected/f27ee3f0-0ef7-4e40-904a-ca944d08f322-kube-api-access-sn5fn\") pod \"f27ee3f0-0ef7-4e40-904a-ca944d08f322\" (UID: \"f27ee3f0-0ef7-4e40-904a-ca944d08f322\") " Sep 07 16:39:28.893431 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:39:28.893402 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f27ee3f0-0ef7-4e40-904a-ca944d08f322-credential-socket\") pod \"f27ee3f0-0ef7-4e40-904a-ca944d08f322\" (UID: \"f27ee3f0-0ef7-4e40-904a-ca944d08f322\") " Sep 07 16:39:28.893431 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:39:28.893422 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f27ee3f0-0ef7-4e40-904a-ca944d08f322-workload-socket\") pod \"f27ee3f0-0ef7-4e40-904a-ca944d08f322\" (UID: \"f27ee3f0-0ef7-4e40-904a-ca944d08f322\") " Sep 07 16:39:28.893704 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:39:28.893448 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f27ee3f0-0ef7-4e40-904a-ca944d08f322-istio-envoy\") pod \"f27ee3f0-0ef7-4e40-904a-ca944d08f322\" (UID: \"f27ee3f0-0ef7-4e40-904a-ca944d08f322\") " Sep 07 16:39:28.893704 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:39:28.893466 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f27ee3f0-0ef7-4e40-904a-ca944d08f322-istio-token\") pod \"f27ee3f0-0ef7-4e40-904a-ca944d08f322\" (UID: \"f27ee3f0-0ef7-4e40-904a-ca944d08f322\") " Sep 07 16:39:28.893704 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:39:28.893521 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f27ee3f0-0ef7-4e40-904a-ca944d08f322-istio-data\") pod \"f27ee3f0-0ef7-4e40-904a-ca944d08f322\" (UID: \"f27ee3f0-0ef7-4e40-904a-ca944d08f322\") " Sep 07 16:39:28.893704 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:39:28.893539 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f27ee3f0-0ef7-4e40-904a-ca944d08f322-workload-certs\") pod \"f27ee3f0-0ef7-4e40-904a-ca944d08f322\" (UID: \"f27ee3f0-0ef7-4e40-904a-ca944d08f322\") " Sep 07 16:39:28.893704 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:39:28.893586 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f27ee3f0-0ef7-4e40-904a-ca944d08f322-istiod-ca-cert\") pod \"f27ee3f0-0ef7-4e40-904a-ca944d08f322\" (UID: \"f27ee3f0-0ef7-4e40-904a-ca944d08f322\") " Sep 07 16:39:28.893704 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:39:28.893614 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f27ee3f0-0ef7-4e40-904a-ca944d08f322-istio-podinfo\") pod \"f27ee3f0-0ef7-4e40-904a-ca944d08f322\" (UID: \"f27ee3f0-0ef7-4e40-904a-ca944d08f322\") " Sep 07 16:39:28.893997 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:39:28.893725 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f27ee3f0-0ef7-4e40-904a-ca944d08f322-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f27ee3f0-0ef7-4e40-904a-ca944d08f322" (UID: "f27ee3f0-0ef7-4e40-904a-ca944d08f322"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 16:39:28.893997 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:39:28.893748 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f27ee3f0-0ef7-4e40-904a-ca944d08f322-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f27ee3f0-0ef7-4e40-904a-ca944d08f322" (UID: "f27ee3f0-0ef7-4e40-904a-ca944d08f322"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 16:39:28.893997 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:39:28.893916 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f27ee3f0-0ef7-4e40-904a-ca944d08f322-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f27ee3f0-0ef7-4e40-904a-ca944d08f322" (UID: "f27ee3f0-0ef7-4e40-904a-ca944d08f322"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 16:39:28.894200 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:39:28.894073 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f27ee3f0-0ef7-4e40-904a-ca944d08f322-credential-socket\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:39:28.894200 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:39:28.894100 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f27ee3f0-0ef7-4e40-904a-ca944d08f322-workload-socket\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:39:28.894200 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:39:28.894073 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f27ee3f0-0ef7-4e40-904a-ca944d08f322-istio-data" (OuterVolumeSpecName: "istio-data") pod "f27ee3f0-0ef7-4e40-904a-ca944d08f322" (UID: "f27ee3f0-0ef7-4e40-904a-ca944d08f322"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 16:39:28.894200 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:39:28.894082 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f27ee3f0-0ef7-4e40-904a-ca944d08f322-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f27ee3f0-0ef7-4e40-904a-ca944d08f322" (UID: "f27ee3f0-0ef7-4e40-904a-ca944d08f322"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 16:39:28.895919 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:39:28.895885 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f27ee3f0-0ef7-4e40-904a-ca944d08f322-kube-api-access-sn5fn" (OuterVolumeSpecName: "kube-api-access-sn5fn") pod "f27ee3f0-0ef7-4e40-904a-ca944d08f322" (UID: "f27ee3f0-0ef7-4e40-904a-ca944d08f322"). InnerVolumeSpecName "kube-api-access-sn5fn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 16:39:28.895919 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:39:28.895912 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f27ee3f0-0ef7-4e40-904a-ca944d08f322-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f27ee3f0-0ef7-4e40-904a-ca944d08f322" (UID: "f27ee3f0-0ef7-4e40-904a-ca944d08f322"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 16:39:28.896354 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:39:28.896338 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f27ee3f0-0ef7-4e40-904a-ca944d08f322-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f27ee3f0-0ef7-4e40-904a-ca944d08f322" (UID: "f27ee3f0-0ef7-4e40-904a-ca944d08f322"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 07 16:39:28.896417 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:39:28.896378 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f27ee3f0-0ef7-4e40-904a-ca944d08f322-istio-token" (OuterVolumeSpecName: "istio-token") pod "f27ee3f0-0ef7-4e40-904a-ca944d08f322" (UID: "f27ee3f0-0ef7-4e40-904a-ca944d08f322"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 16:39:28.949957 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:39:28.949920 2580 generic.go:356] "Generic (PLEG): container finished" podID="f27ee3f0-0ef7-4e40-904a-ca944d08f322" containerID="91cccbe5b6301b69db032e6a666ee4782729073e6c750e6fa951256c9fb9fee2" exitCode=0 Sep 07 16:39:28.950160 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:39:28.949974 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8" Sep 07 16:39:28.950160 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:39:28.950014 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8" event={"ID":"f27ee3f0-0ef7-4e40-904a-ca944d08f322","Type":"ContainerDied","Data":"91cccbe5b6301b69db032e6a666ee4782729073e6c750e6fa951256c9fb9fee2"} Sep 07 16:39:28.950160 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:39:28.950101 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8" event={"ID":"f27ee3f0-0ef7-4e40-904a-ca944d08f322","Type":"ContainerDied","Data":"50d5a106fd89d0782d3755cc3236f3ad95931a895943f39af54b6d8c9a83e1f0"} Sep 07 16:39:28.950160 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:39:28.950125 2580 scope.go:117] "RemoveContainer" containerID="91cccbe5b6301b69db032e6a666ee4782729073e6c750e6fa951256c9fb9fee2" Sep 07 16:39:28.959688 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:39:28.959673 2580 scope.go:117] "RemoveContainer" containerID="1d957e8baa80e62a8246fe26bb37ff8a70fe44483c606fd14fefecad1c40177a" Sep 07 16:39:28.968647 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:39:28.968620 2580 scope.go:117] "RemoveContainer" containerID="91cccbe5b6301b69db032e6a666ee4782729073e6c750e6fa951256c9fb9fee2" Sep 07 16:39:28.968977 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:39:28.968955 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91cccbe5b6301b69db032e6a666ee4782729073e6c750e6fa951256c9fb9fee2\": container with ID starting with 91cccbe5b6301b69db032e6a666ee4782729073e6c750e6fa951256c9fb9fee2 not found: ID does not exist" containerID="91cccbe5b6301b69db032e6a666ee4782729073e6c750e6fa951256c9fb9fee2" Sep 07 16:39:28.969099 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:39:28.968991 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91cccbe5b6301b69db032e6a666ee4782729073e6c750e6fa951256c9fb9fee2"} err="failed to get container status \"91cccbe5b6301b69db032e6a666ee4782729073e6c750e6fa951256c9fb9fee2\": rpc error: code = NotFound desc = could not find container \"91cccbe5b6301b69db032e6a666ee4782729073e6c750e6fa951256c9fb9fee2\": container with ID starting with 91cccbe5b6301b69db032e6a666ee4782729073e6c750e6fa951256c9fb9fee2 not found: ID does not exist" Sep 07 16:39:28.969099 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:39:28.969016 2580 scope.go:117] "RemoveContainer" containerID="1d957e8baa80e62a8246fe26bb37ff8a70fe44483c606fd14fefecad1c40177a" Sep 07 16:39:28.969332 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:39:28.969316 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d957e8baa80e62a8246fe26bb37ff8a70fe44483c606fd14fefecad1c40177a\": container with ID starting with 1d957e8baa80e62a8246fe26bb37ff8a70fe44483c606fd14fefecad1c40177a not found: ID does not exist" containerID="1d957e8baa80e62a8246fe26bb37ff8a70fe44483c606fd14fefecad1c40177a" Sep 07 16:39:28.969381 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:39:28.969340 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d957e8baa80e62a8246fe26bb37ff8a70fe44483c606fd14fefecad1c40177a"} err="failed to get container status \"1d957e8baa80e62a8246fe26bb37ff8a70fe44483c606fd14fefecad1c40177a\": rpc error: code = NotFound desc = could not find container \"1d957e8baa80e62a8246fe26bb37ff8a70fe44483c606fd14fefecad1c40177a\": container with ID starting with 1d957e8baa80e62a8246fe26bb37ff8a70fe44483c606fd14fefecad1c40177a not found: ID does not exist" Sep 07 16:39:28.970211 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:39:28.970193 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8"] Sep 07 16:39:28.975817 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:39:28.975795 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-2ff801c7-openshift-default-5cf78565bd-lgnt8"] Sep 07 16:39:28.994757 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:39:28.994730 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f27ee3f0-0ef7-4e40-904a-ca944d08f322-istiod-ca-cert\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:39:28.994757 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:39:28.994757 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f27ee3f0-0ef7-4e40-904a-ca944d08f322-istio-podinfo\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:39:28.994919 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:39:28.994771 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sn5fn\" (UniqueName: \"kubernetes.io/projected/f27ee3f0-0ef7-4e40-904a-ca944d08f322-kube-api-access-sn5fn\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:39:28.994919 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:39:28.994785 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f27ee3f0-0ef7-4e40-904a-ca944d08f322-istio-envoy\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:39:28.994919 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:39:28.994796 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f27ee3f0-0ef7-4e40-904a-ca944d08f322-istio-token\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:39:28.994919 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:39:28.994807 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f27ee3f0-0ef7-4e40-904a-ca944d08f322-istio-data\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:39:28.994919 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:39:28.994818 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f27ee3f0-0ef7-4e40-904a-ca944d08f322-workload-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:39:29.417904 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:39:29.417862 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f27ee3f0-0ef7-4e40-904a-ca944d08f322" path="/var/lib/kubelet/pods/f27ee3f0-0ef7-4e40-904a-ca944d08f322/volumes" Sep 07 16:39:31.578352 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:39:31.578299 2580 scope.go:117] "RemoveContainer" containerID="3ba80e83186f6decfb2bd76b346ca5e89281c593a03f6a5933f77b03b27a7198" Sep 07 16:39:31.586998 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:39:31.586978 2580 scope.go:117] "RemoveContainer" containerID="d582e5fd6fba39ee1dae0f6455cd6097f788d90739a32791531a31d44c42efb4" Sep 07 16:39:31.594850 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:39:31.594829 2580 scope.go:117] "RemoveContainer" containerID="0b2bc20721af3fabbdf3234324202c60ef21be12c10fddbf565fb4404516a5ed" Sep 07 16:40:17.265565 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:17.265525 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj"] Sep 07 16:40:17.266230 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:17.266210 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="786fae7b-5296-4bf2-9270-312191ed73b2" containerName="revoke-keys" Sep 07 16:40:17.266282 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:17.266236 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="786fae7b-5296-4bf2-9270-312191ed73b2" containerName="revoke-keys" Sep 07 16:40:17.266282 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:17.266265 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12647717-ec9c-4960-ac57-a0ee0aa001b6" containerName="payload-processing" Sep 07 16:40:17.266282 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:17.266273 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="12647717-ec9c-4960-ac57-a0ee0aa001b6" containerName="payload-processing" Sep 07 16:40:17.266406 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:17.266283 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a98227c-12fd-4da2-b170-2f682d0c4818" containerName="payload-pre-processing" Sep 07 16:40:17.266406 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:17.266292 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a98227c-12fd-4da2-b170-2f682d0c4818" containerName="payload-pre-processing" Sep 07 16:40:17.266406 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:17.266308 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f27ee3f0-0ef7-4e40-904a-ca944d08f322" containerName="istio-proxy" Sep 07 16:40:17.266406 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:17.266316 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f27ee3f0-0ef7-4e40-904a-ca944d08f322" containerName="istio-proxy" Sep 07 16:40:17.266406 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:17.266326 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f27ee3f0-0ef7-4e40-904a-ca944d08f322" containerName="istio-proxy" Sep 07 16:40:17.266406 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:17.266335 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f27ee3f0-0ef7-4e40-904a-ca944d08f322" containerName="istio-proxy" Sep 07 16:40:17.266589 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:17.266445 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a98227c-12fd-4da2-b170-2f682d0c4818" containerName="payload-pre-processing" Sep 07 16:40:17.266589 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:17.266455 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="12647717-ec9c-4960-ac57-a0ee0aa001b6" containerName="payload-processing" Sep 07 16:40:17.266589 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:17.266466 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f27ee3f0-0ef7-4e40-904a-ca944d08f322" containerName="istio-proxy" Sep 07 16:40:17.266589 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:17.266476 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f27ee3f0-0ef7-4e40-904a-ca944d08f322" containerName="istio-proxy" Sep 07 16:40:17.266589 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:17.266486 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="786fae7b-5296-4bf2-9270-312191ed73b2" containerName="revoke-keys" Sep 07 16:40:17.269913 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:17.269893 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj" Sep 07 16:40:17.271998 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:17.271975 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-8ba4d4-openshift-default-dockercfg-lhk2d\"" Sep 07 16:40:17.281374 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:17.281345 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj"] Sep 07 16:40:17.345511 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:17.345470 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/515038a5-c027-4894-ac05-08db9c29f3c6-istio-podinfo\") pod \"e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj\" (UID: \"515038a5-c027-4894-ac05-08db9c29f3c6\") " pod="openshift-ingress/e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj" Sep 07 16:40:17.345697 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:17.345552 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/515038a5-c027-4894-ac05-08db9c29f3c6-workload-socket\") pod \"e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj\" (UID: \"515038a5-c027-4894-ac05-08db9c29f3c6\") " pod="openshift-ingress/e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj" Sep 07 16:40:17.345697 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:17.345630 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/515038a5-c027-4894-ac05-08db9c29f3c6-istio-envoy\") pod \"e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj\" (UID: \"515038a5-c027-4894-ac05-08db9c29f3c6\") " pod="openshift-ingress/e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj" Sep 07 16:40:17.345697 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:17.345676 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75m42\" (UniqueName: \"kubernetes.io/projected/515038a5-c027-4894-ac05-08db9c29f3c6-kube-api-access-75m42\") pod \"e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj\" (UID: \"515038a5-c027-4894-ac05-08db9c29f3c6\") " pod="openshift-ingress/e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj" Sep 07 16:40:17.345808 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:17.345741 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/515038a5-c027-4894-ac05-08db9c29f3c6-credential-socket\") pod \"e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj\" (UID: \"515038a5-c027-4894-ac05-08db9c29f3c6\") " pod="openshift-ingress/e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj" Sep 07 16:40:17.345808 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:17.345771 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/515038a5-c027-4894-ac05-08db9c29f3c6-workload-certs\") pod \"e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj\" (UID: \"515038a5-c027-4894-ac05-08db9c29f3c6\") " pod="openshift-ingress/e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj" Sep 07 16:40:17.345808 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:17.345788 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/515038a5-c027-4894-ac05-08db9c29f3c6-istio-data\") pod \"e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj\" (UID: \"515038a5-c027-4894-ac05-08db9c29f3c6\") " pod="openshift-ingress/e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj" Sep 07 16:40:17.345808 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:17.345806 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/515038a5-c027-4894-ac05-08db9c29f3c6-istiod-ca-cert\") pod \"e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj\" (UID: \"515038a5-c027-4894-ac05-08db9c29f3c6\") " pod="openshift-ingress/e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj" Sep 07 16:40:17.345946 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:17.345825 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/515038a5-c027-4894-ac05-08db9c29f3c6-istio-token\") pod \"e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj\" (UID: \"515038a5-c027-4894-ac05-08db9c29f3c6\") " pod="openshift-ingress/e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj" Sep 07 16:40:17.446428 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:17.446369 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/515038a5-c027-4894-ac05-08db9c29f3c6-istio-envoy\") pod \"e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj\" (UID: \"515038a5-c027-4894-ac05-08db9c29f3c6\") " pod="openshift-ingress/e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj" Sep 07 16:40:17.446428 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:17.446438 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75m42\" (UniqueName: \"kubernetes.io/projected/515038a5-c027-4894-ac05-08db9c29f3c6-kube-api-access-75m42\") pod \"e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj\" (UID: \"515038a5-c027-4894-ac05-08db9c29f3c6\") " pod="openshift-ingress/e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj" Sep 07 16:40:17.446698 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:17.446480 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/515038a5-c027-4894-ac05-08db9c29f3c6-credential-socket\") pod \"e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj\" (UID: \"515038a5-c027-4894-ac05-08db9c29f3c6\") " pod="openshift-ingress/e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj" Sep 07 16:40:17.446698 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:17.446499 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/515038a5-c027-4894-ac05-08db9c29f3c6-workload-certs\") pod \"e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj\" (UID: \"515038a5-c027-4894-ac05-08db9c29f3c6\") " pod="openshift-ingress/e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj" Sep 07 16:40:17.446698 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:17.446522 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/515038a5-c027-4894-ac05-08db9c29f3c6-istio-data\") pod \"e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj\" (UID: \"515038a5-c027-4894-ac05-08db9c29f3c6\") " pod="openshift-ingress/e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj" Sep 07 16:40:17.446698 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:17.446552 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/515038a5-c027-4894-ac05-08db9c29f3c6-istiod-ca-cert\") pod \"e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj\" (UID: \"515038a5-c027-4894-ac05-08db9c29f3c6\") " pod="openshift-ingress/e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj" Sep 07 16:40:17.446698 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:17.446579 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/515038a5-c027-4894-ac05-08db9c29f3c6-istio-token\") pod \"e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj\" (UID: \"515038a5-c027-4894-ac05-08db9c29f3c6\") " pod="openshift-ingress/e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj" Sep 07 16:40:17.446698 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:17.446628 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/515038a5-c027-4894-ac05-08db9c29f3c6-istio-podinfo\") pod \"e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj\" (UID: \"515038a5-c027-4894-ac05-08db9c29f3c6\") " pod="openshift-ingress/e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj" Sep 07 16:40:17.447017 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:17.446712 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/515038a5-c027-4894-ac05-08db9c29f3c6-workload-socket\") pod \"e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj\" (UID: \"515038a5-c027-4894-ac05-08db9c29f3c6\") " pod="openshift-ingress/e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj" Sep 07 16:40:17.447017 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:17.446897 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/515038a5-c027-4894-ac05-08db9c29f3c6-credential-socket\") pod \"e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj\" (UID: \"515038a5-c027-4894-ac05-08db9c29f3c6\") " pod="openshift-ingress/e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj" Sep 07 16:40:17.447240 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:17.447216 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/515038a5-c027-4894-ac05-08db9c29f3c6-workload-certs\") pod \"e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj\" (UID: \"515038a5-c027-4894-ac05-08db9c29f3c6\") " pod="openshift-ingress/e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj" Sep 07 16:40:17.447301 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:17.447245 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/515038a5-c027-4894-ac05-08db9c29f3c6-istio-data\") pod \"e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj\" (UID: \"515038a5-c027-4894-ac05-08db9c29f3c6\") " pod="openshift-ingress/e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj" Sep 07 16:40:17.447301 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:17.447274 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/515038a5-c027-4894-ac05-08db9c29f3c6-workload-socket\") pod \"e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj\" (UID: \"515038a5-c027-4894-ac05-08db9c29f3c6\") " pod="openshift-ingress/e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj" Sep 07 16:40:17.447409 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:17.447306 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/515038a5-c027-4894-ac05-08db9c29f3c6-istiod-ca-cert\") pod \"e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj\" (UID: \"515038a5-c027-4894-ac05-08db9c29f3c6\") " pod="openshift-ingress/e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj" Sep 07 16:40:17.448977 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:17.448960 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/515038a5-c027-4894-ac05-08db9c29f3c6-istio-envoy\") pod \"e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj\" (UID: \"515038a5-c027-4894-ac05-08db9c29f3c6\") " pod="openshift-ingress/e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj" Sep 07 16:40:17.449526 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:17.449510 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/515038a5-c027-4894-ac05-08db9c29f3c6-istio-podinfo\") pod \"e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj\" (UID: \"515038a5-c027-4894-ac05-08db9c29f3c6\") " pod="openshift-ingress/e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj" Sep 07 16:40:17.454200 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:17.454175 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/515038a5-c027-4894-ac05-08db9c29f3c6-istio-token\") pod \"e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj\" (UID: \"515038a5-c027-4894-ac05-08db9c29f3c6\") " pod="openshift-ingress/e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj" Sep 07 16:40:17.455129 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:17.455100 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75m42\" (UniqueName: \"kubernetes.io/projected/515038a5-c027-4894-ac05-08db9c29f3c6-kube-api-access-75m42\") pod \"e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj\" (UID: \"515038a5-c027-4894-ac05-08db9c29f3c6\") " pod="openshift-ingress/e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj" Sep 07 16:40:17.583843 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:17.583751 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj" Sep 07 16:40:17.870816 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:17.870787 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj"] Sep 07 16:40:17.873388 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:40:17.873354 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod515038a5_c027_4894_ac05_08db9c29f3c6.slice/crio-100f60558bfc07a0e45ee4e3ff155243829dbb0846b6f2eab1f19695529dcae7 WatchSource:0}: Error finding container 100f60558bfc07a0e45ee4e3ff155243829dbb0846b6f2eab1f19695529dcae7: Status 404 returned error can't find the container with id 100f60558bfc07a0e45ee4e3ff155243829dbb0846b6f2eab1f19695529dcae7 Sep 07 16:40:17.876121 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:17.876090 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:40:17.876189 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:17.876163 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:40:17.876227 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:17.876189 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:40:18.180544 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:18.180451 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj" event={"ID":"515038a5-c027-4894-ac05-08db9c29f3c6","Type":"ContainerStarted","Data":"f78b03fa150d33db9a70dd9112a274782f8c7e65c11b06f7faa20e3e2a59cbb0"} Sep 07 16:40:18.180544 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:18.180501 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj" event={"ID":"515038a5-c027-4894-ac05-08db9c29f3c6","Type":"ContainerStarted","Data":"100f60558bfc07a0e45ee4e3ff155243829dbb0846b6f2eab1f19695529dcae7"} Sep 07 16:40:18.199822 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:18.199759 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj" podStartSLOduration=1.199742183 podStartE2EDuration="1.199742183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:40:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:40:18.197248078 +0000 UTC m=+1247.232471171" watchObservedRunningTime="2026-09-07 16:40:18.199742183 +0000 UTC m=+1247.234965253" Sep 07 16:40:18.583978 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:18.583939 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj" Sep 07 16:40:18.589410 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:18.589387 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj" Sep 07 16:40:19.185166 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:19.185132 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj" Sep 07 16:40:19.186198 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:19.186178 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj" Sep 07 16:40:21.785291 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:21.785250 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8ba4d4-7c7b47469d-zw2f7"] Sep 07 16:40:21.788983 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:21.788957 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8ba4d4-7c7b47469d-zw2f7" Sep 07 16:40:21.791467 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:21.791444 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-8ba4d4\"" Sep 07 16:40:21.791601 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:21.791495 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-8ba4d4-dockercfg-nd6fz\"" Sep 07 16:40:21.799059 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:21.799009 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8ba4d4-7c7b47469d-zw2f7"] Sep 07 16:40:21.896166 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:21.896126 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwtmw\" (UniqueName: \"kubernetes.io/projected/0694cdbd-ea07-4980-a668-66bca8cb7b2f-kube-api-access-fwtmw\") pod \"payload-pre-processing-e2e-api-a-8ba4d4-7c7b47469d-zw2f7\" (UID: \"0694cdbd-ea07-4980-a668-66bca8cb7b2f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8ba4d4-7c7b47469d-zw2f7" Sep 07 16:40:21.896352 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:21.896180 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0694cdbd-ea07-4980-a668-66bca8cb7b2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-8ba4d4-7c7b47469d-zw2f7\" (UID: \"0694cdbd-ea07-4980-a668-66bca8cb7b2f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8ba4d4-7c7b47469d-zw2f7" Sep 07 16:40:21.997400 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:21.997364 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwtmw\" (UniqueName: \"kubernetes.io/projected/0694cdbd-ea07-4980-a668-66bca8cb7b2f-kube-api-access-fwtmw\") pod \"payload-pre-processing-e2e-api-a-8ba4d4-7c7b47469d-zw2f7\" (UID: \"0694cdbd-ea07-4980-a668-66bca8cb7b2f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8ba4d4-7c7b47469d-zw2f7" Sep 07 16:40:21.997585 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:21.997407 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0694cdbd-ea07-4980-a668-66bca8cb7b2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-8ba4d4-7c7b47469d-zw2f7\" (UID: \"0694cdbd-ea07-4980-a668-66bca8cb7b2f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8ba4d4-7c7b47469d-zw2f7" Sep 07 16:40:21.998039 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:21.998009 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0694cdbd-ea07-4980-a668-66bca8cb7b2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-8ba4d4-7c7b47469d-zw2f7\" (UID: \"0694cdbd-ea07-4980-a668-66bca8cb7b2f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8ba4d4-7c7b47469d-zw2f7" Sep 07 16:40:22.019395 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:22.019363 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwtmw\" (UniqueName: \"kubernetes.io/projected/0694cdbd-ea07-4980-a668-66bca8cb7b2f-kube-api-access-fwtmw\") pod \"payload-pre-processing-e2e-api-a-8ba4d4-7c7b47469d-zw2f7\" (UID: \"0694cdbd-ea07-4980-a668-66bca8cb7b2f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8ba4d4-7c7b47469d-zw2f7" Sep 07 16:40:22.100505 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:22.100413 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8ba4d4-7c7b47469d-zw2f7" Sep 07 16:40:22.369739 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:22.369709 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8ba4d4-7c7b47469d-zw2f7"] Sep 07 16:40:22.372178 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:40:22.372149 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0694cdbd_ea07_4980_a668_66bca8cb7b2f.slice/crio-8840f0b20c0775d9dbde4780ea6eb0b55b9346d5836f6d7e65e6246f911fe5e7 WatchSource:0}: Error finding container 8840f0b20c0775d9dbde4780ea6eb0b55b9346d5836f6d7e65e6246f911fe5e7: Status 404 returned error can't find the container with id 8840f0b20c0775d9dbde4780ea6eb0b55b9346d5836f6d7e65e6246f911fe5e7 Sep 07 16:40:23.204945 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:23.204909 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8ba4d4-7c7b47469d-zw2f7" event={"ID":"0694cdbd-ea07-4980-a668-66bca8cb7b2f","Type":"ContainerStarted","Data":"651676560eb400174f26f9445220c6b03f0f07427793516ed0523f602130fe6d"} Sep 07 16:40:23.205367 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:23.204960 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8ba4d4-7c7b47469d-zw2f7" event={"ID":"0694cdbd-ea07-4980-a668-66bca8cb7b2f","Type":"ContainerStarted","Data":"8840f0b20c0775d9dbde4780ea6eb0b55b9346d5836f6d7e65e6246f911fe5e7"} Sep 07 16:40:23.205367 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:23.205065 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8ba4d4-7c7b47469d-zw2f7" Sep 07 16:40:23.224070 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:23.224007 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8ba4d4-7c7b47469d-zw2f7" podStartSLOduration=2.223991193 podStartE2EDuration="2.223991193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:40:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:40:23.221548654 +0000 UTC m=+1252.256771725" watchObservedRunningTime="2026-09-07 16:40:23.223991193 +0000 UTC m=+1252.259214263" Sep 07 16:40:31.631934 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:31.631889 2580 scope.go:117] "RemoveContainer" containerID="188d01da977d39ac9a72bd18aaa104e38e5e18adae3256788b5d8e4adce510e3" Sep 07 16:40:34.212812 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:34.212772 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8ba4d4-7c7b47469d-zw2f7" Sep 07 16:40:39.288292 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:39.288194 2580 generic.go:356] "Generic (PLEG): container finished" podID="515038a5-c027-4894-ac05-08db9c29f3c6" containerID="f78b03fa150d33db9a70dd9112a274782f8c7e65c11b06f7faa20e3e2a59cbb0" exitCode=0 Sep 07 16:40:39.288292 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:39.288270 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj" event={"ID":"515038a5-c027-4894-ac05-08db9c29f3c6","Type":"ContainerDied","Data":"f78b03fa150d33db9a70dd9112a274782f8c7e65c11b06f7faa20e3e2a59cbb0"} Sep 07 16:40:39.288782 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:39.288766 2580 scope.go:117] "RemoveContainer" containerID="f78b03fa150d33db9a70dd9112a274782f8c7e65c11b06f7faa20e3e2a59cbb0" Sep 07 16:40:39.289885 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:39.289851 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:40:39.289991 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:39.289930 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:40:39.289991 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:39.289976 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:40:40.294645 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:40.294607 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj" event={"ID":"515038a5-c027-4894-ac05-08db9c29f3c6","Type":"ContainerStarted","Data":"f78f38f6dffeb7b521d8015ed227157e1565330417fa8924a99935c6c19be415"} Sep 07 16:40:40.584533 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:40.584447 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj" Sep 07 16:40:40.584709 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:40.584685 2580 patch_prober.go:28] interesting pod/e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Sep 07 16:40:40.584752 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:40.584737 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj" podUID="515038a5-c027-4894-ac05-08db9c29f3c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Sep 07 16:40:41.584404 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:41.584362 2580 patch_prober.go:28] interesting pod/e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Sep 07 16:40:41.584787 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:41.584427 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj" podUID="515038a5-c027-4894-ac05-08db9c29f3c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Sep 07 16:40:42.584890 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:42.584851 2580 patch_prober.go:28] interesting pod/e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Sep 07 16:40:42.585329 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:42.584928 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj" podUID="515038a5-c027-4894-ac05-08db9c29f3c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Sep 07 16:40:43.584363 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:43.584326 2580 patch_prober.go:28] interesting pod/e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Sep 07 16:40:43.584547 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:43.584386 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj" podUID="515038a5-c027-4894-ac05-08db9c29f3c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Sep 07 16:40:44.584157 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:44.584120 2580 patch_prober.go:28] interesting pod/e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Sep 07 16:40:44.584626 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:44.584187 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj" podUID="515038a5-c027-4894-ac05-08db9c29f3c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Sep 07 16:40:45.701140 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:45.701103 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4"] Sep 07 16:40:45.704767 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:45.704742 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4" Sep 07 16:40:45.708216 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:45.707344 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-d7b6a5-openshift-default-dockercfg-n9dsh\"" Sep 07 16:40:45.718076 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:45.718016 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4"] Sep 07 16:40:45.741750 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:45.741704 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed56d248-38d2-42d3-bb8c-31b9c14f3c2d-credential-socket\") pod \"e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4\" (UID: \"ed56d248-38d2-42d3-bb8c-31b9c14f3c2d\") " pod="openshift-ingress/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4" Sep 07 16:40:45.742123 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:45.741762 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed56d248-38d2-42d3-bb8c-31b9c14f3c2d-istio-podinfo\") pod \"e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4\" (UID: \"ed56d248-38d2-42d3-bb8c-31b9c14f3c2d\") " pod="openshift-ingress/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4" Sep 07 16:40:45.742123 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:45.741792 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed56d248-38d2-42d3-bb8c-31b9c14f3c2d-workload-certs\") pod \"e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4\" (UID: \"ed56d248-38d2-42d3-bb8c-31b9c14f3c2d\") " pod="openshift-ingress/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4" Sep 07 16:40:45.742123 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:45.741828 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed56d248-38d2-42d3-bb8c-31b9c14f3c2d-istio-envoy\") pod \"e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4\" (UID: \"ed56d248-38d2-42d3-bb8c-31b9c14f3c2d\") " pod="openshift-ingress/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4" Sep 07 16:40:45.742123 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:45.741863 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed56d248-38d2-42d3-bb8c-31b9c14f3c2d-workload-socket\") pod \"e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4\" (UID: \"ed56d248-38d2-42d3-bb8c-31b9c14f3c2d\") " pod="openshift-ingress/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4" Sep 07 16:40:45.742123 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:45.741899 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5l7x\" (UniqueName: \"kubernetes.io/projected/ed56d248-38d2-42d3-bb8c-31b9c14f3c2d-kube-api-access-m5l7x\") pod \"e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4\" (UID: \"ed56d248-38d2-42d3-bb8c-31b9c14f3c2d\") " pod="openshift-ingress/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4" Sep 07 16:40:45.742123 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:45.741947 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed56d248-38d2-42d3-bb8c-31b9c14f3c2d-istio-token\") pod \"e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4\" (UID: \"ed56d248-38d2-42d3-bb8c-31b9c14f3c2d\") " pod="openshift-ingress/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4" Sep 07 16:40:45.742123 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:45.742005 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed56d248-38d2-42d3-bb8c-31b9c14f3c2d-istio-data\") pod \"e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4\" (UID: \"ed56d248-38d2-42d3-bb8c-31b9c14f3c2d\") " pod="openshift-ingress/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4" Sep 07 16:40:45.742705 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:45.742157 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed56d248-38d2-42d3-bb8c-31b9c14f3c2d-istiod-ca-cert\") pod \"e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4\" (UID: \"ed56d248-38d2-42d3-bb8c-31b9c14f3c2d\") " pod="openshift-ingress/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4" Sep 07 16:40:45.787935 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:45.787904 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj" Sep 07 16:40:45.788932 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:45.788906 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj" Sep 07 16:40:45.791398 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:45.791372 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj" Sep 07 16:40:45.843303 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:45.843093 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed56d248-38d2-42d3-bb8c-31b9c14f3c2d-istio-token\") pod \"e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4\" (UID: \"ed56d248-38d2-42d3-bb8c-31b9c14f3c2d\") " pod="openshift-ingress/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4" Sep 07 16:40:45.843303 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:45.843175 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed56d248-38d2-42d3-bb8c-31b9c14f3c2d-istio-data\") pod \"e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4\" (UID: \"ed56d248-38d2-42d3-bb8c-31b9c14f3c2d\") " pod="openshift-ingress/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4" Sep 07 16:40:45.843303 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:45.843282 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed56d248-38d2-42d3-bb8c-31b9c14f3c2d-istiod-ca-cert\") pod \"e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4\" (UID: \"ed56d248-38d2-42d3-bb8c-31b9c14f3c2d\") " pod="openshift-ingress/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4" Sep 07 16:40:45.843696 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:45.843346 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed56d248-38d2-42d3-bb8c-31b9c14f3c2d-credential-socket\") pod \"e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4\" (UID: \"ed56d248-38d2-42d3-bb8c-31b9c14f3c2d\") " pod="openshift-ingress/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4" Sep 07 16:40:45.843696 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:45.843368 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed56d248-38d2-42d3-bb8c-31b9c14f3c2d-istio-podinfo\") pod \"e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4\" (UID: \"ed56d248-38d2-42d3-bb8c-31b9c14f3c2d\") " pod="openshift-ingress/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4" Sep 07 16:40:45.843696 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:45.843390 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed56d248-38d2-42d3-bb8c-31b9c14f3c2d-workload-certs\") pod \"e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4\" (UID: \"ed56d248-38d2-42d3-bb8c-31b9c14f3c2d\") " pod="openshift-ingress/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4" Sep 07 16:40:45.843696 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:45.843424 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed56d248-38d2-42d3-bb8c-31b9c14f3c2d-istio-envoy\") pod \"e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4\" (UID: \"ed56d248-38d2-42d3-bb8c-31b9c14f3c2d\") " pod="openshift-ingress/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4" Sep 07 16:40:45.843696 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:45.843458 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed56d248-38d2-42d3-bb8c-31b9c14f3c2d-workload-socket\") pod \"e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4\" (UID: \"ed56d248-38d2-42d3-bb8c-31b9c14f3c2d\") " pod="openshift-ingress/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4" Sep 07 16:40:45.843696 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:45.843504 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5l7x\" (UniqueName: \"kubernetes.io/projected/ed56d248-38d2-42d3-bb8c-31b9c14f3c2d-kube-api-access-m5l7x\") pod \"e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4\" (UID: \"ed56d248-38d2-42d3-bb8c-31b9c14f3c2d\") " pod="openshift-ingress/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4" Sep 07 16:40:45.844453 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:45.844428 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed56d248-38d2-42d3-bb8c-31b9c14f3c2d-credential-socket\") pod \"e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4\" (UID: \"ed56d248-38d2-42d3-bb8c-31b9c14f3c2d\") " pod="openshift-ingress/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4" Sep 07 16:40:45.844593 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:45.844449 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed56d248-38d2-42d3-bb8c-31b9c14f3c2d-istio-data\") pod \"e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4\" (UID: \"ed56d248-38d2-42d3-bb8c-31b9c14f3c2d\") " pod="openshift-ingress/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4" Sep 07 16:40:45.846086 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:45.845380 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed56d248-38d2-42d3-bb8c-31b9c14f3c2d-workload-socket\") pod \"e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4\" (UID: \"ed56d248-38d2-42d3-bb8c-31b9c14f3c2d\") " pod="openshift-ingress/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4" Sep 07 16:40:45.846086 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:45.845530 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed56d248-38d2-42d3-bb8c-31b9c14f3c2d-istiod-ca-cert\") pod \"e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4\" (UID: \"ed56d248-38d2-42d3-bb8c-31b9c14f3c2d\") " pod="openshift-ingress/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4" Sep 07 16:40:45.846086 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:45.845982 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed56d248-38d2-42d3-bb8c-31b9c14f3c2d-workload-certs\") pod \"e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4\" (UID: \"ed56d248-38d2-42d3-bb8c-31b9c14f3c2d\") " pod="openshift-ingress/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4" Sep 07 16:40:45.849657 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:45.849371 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed56d248-38d2-42d3-bb8c-31b9c14f3c2d-istio-envoy\") pod \"e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4\" (UID: \"ed56d248-38d2-42d3-bb8c-31b9c14f3c2d\") " pod="openshift-ingress/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4" Sep 07 16:40:45.854042 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:45.850007 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed56d248-38d2-42d3-bb8c-31b9c14f3c2d-istio-podinfo\") pod \"e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4\" (UID: \"ed56d248-38d2-42d3-bb8c-31b9c14f3c2d\") " pod="openshift-ingress/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4" Sep 07 16:40:45.861079 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:45.859005 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed56d248-38d2-42d3-bb8c-31b9c14f3c2d-istio-token\") pod \"e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4\" (UID: \"ed56d248-38d2-42d3-bb8c-31b9c14f3c2d\") " pod="openshift-ingress/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4" Sep 07 16:40:45.861079 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:45.859729 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5l7x\" (UniqueName: \"kubernetes.io/projected/ed56d248-38d2-42d3-bb8c-31b9c14f3c2d-kube-api-access-m5l7x\") pod \"e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4\" (UID: \"ed56d248-38d2-42d3-bb8c-31b9c14f3c2d\") " pod="openshift-ingress/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4" Sep 07 16:40:46.023326 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:46.023244 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4" Sep 07 16:40:46.306766 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:46.306705 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4"] Sep 07 16:40:46.309118 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:40:46.309087 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded56d248_38d2_42d3_bb8c_31b9c14f3c2d.slice/crio-f659488c0cfc829204632b66b95e64a12cafcf2d887412ae138ec1f6148043b0 WatchSource:0}: Error finding container f659488c0cfc829204632b66b95e64a12cafcf2d887412ae138ec1f6148043b0: Status 404 returned error can't find the container with id f659488c0cfc829204632b66b95e64a12cafcf2d887412ae138ec1f6148043b0 Sep 07 16:40:46.311379 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:46.311343 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:40:46.311506 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:46.311412 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:40:46.311506 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:46.311459 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:40:46.341248 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:46.341217 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4" event={"ID":"ed56d248-38d2-42d3-bb8c-31b9c14f3c2d","Type":"ContainerStarted","Data":"f659488c0cfc829204632b66b95e64a12cafcf2d887412ae138ec1f6148043b0"} Sep 07 16:40:47.347938 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:47.347897 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4" event={"ID":"ed56d248-38d2-42d3-bb8c-31b9c14f3c2d","Type":"ContainerStarted","Data":"1a8a0b5d9bebc753f087e12f86bb9e8a42af8e12c3bfade71e6125b63d4c9282"} Sep 07 16:40:47.367398 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:47.367353 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4" podStartSLOduration=2.367337102 podStartE2EDuration="2.367337102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:40:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:40:47.364833866 +0000 UTC m=+1276.400056958" watchObservedRunningTime="2026-09-07 16:40:47.367337102 +0000 UTC m=+1276.402560173" Sep 07 16:40:48.024007 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:48.023968 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4" Sep 07 16:40:48.029117 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:48.029091 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4" Sep 07 16:40:48.353662 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:48.353582 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4" Sep 07 16:40:48.354578 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:48.354560 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4" Sep 07 16:40:51.052420 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:51.052385 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d7b6a5-978c477db-89wrk"] Sep 07 16:40:51.055049 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:51.055001 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d7b6a5-978c477db-89wrk" Sep 07 16:40:51.057050 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:51.057010 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-d7b6a5\"" Sep 07 16:40:51.057149 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:51.057016 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-d7b6a5-dockercfg-h9pbd\"" Sep 07 16:40:51.064664 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:51.064641 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d7b6a5-978c477db-89wrk"] Sep 07 16:40:51.097812 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:51.097779 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b2e3e0a-f07d-4c05-b56b-46a35a9bd810-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d7b6a5-978c477db-89wrk\" (UID: \"9b2e3e0a-f07d-4c05-b56b-46a35a9bd810\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d7b6a5-978c477db-89wrk" Sep 07 16:40:51.097991 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:51.097867 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgvl5\" (UniqueName: \"kubernetes.io/projected/9b2e3e0a-f07d-4c05-b56b-46a35a9bd810-kube-api-access-qgvl5\") pod \"payload-pre-processing-e2e-api-b-d7b6a5-978c477db-89wrk\" (UID: \"9b2e3e0a-f07d-4c05-b56b-46a35a9bd810\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d7b6a5-978c477db-89wrk" Sep 07 16:40:51.198564 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:51.198517 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b2e3e0a-f07d-4c05-b56b-46a35a9bd810-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d7b6a5-978c477db-89wrk\" (UID: \"9b2e3e0a-f07d-4c05-b56b-46a35a9bd810\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d7b6a5-978c477db-89wrk" Sep 07 16:40:51.198778 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:51.198588 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgvl5\" (UniqueName: \"kubernetes.io/projected/9b2e3e0a-f07d-4c05-b56b-46a35a9bd810-kube-api-access-qgvl5\") pod \"payload-pre-processing-e2e-api-b-d7b6a5-978c477db-89wrk\" (UID: \"9b2e3e0a-f07d-4c05-b56b-46a35a9bd810\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d7b6a5-978c477db-89wrk" Sep 07 16:40:51.199360 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:51.199334 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b2e3e0a-f07d-4c05-b56b-46a35a9bd810-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d7b6a5-978c477db-89wrk\" (UID: \"9b2e3e0a-f07d-4c05-b56b-46a35a9bd810\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d7b6a5-978c477db-89wrk" Sep 07 16:40:51.206920 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:51.206886 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgvl5\" (UniqueName: \"kubernetes.io/projected/9b2e3e0a-f07d-4c05-b56b-46a35a9bd810-kube-api-access-qgvl5\") pod \"payload-pre-processing-e2e-api-b-d7b6a5-978c477db-89wrk\" (UID: \"9b2e3e0a-f07d-4c05-b56b-46a35a9bd810\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d7b6a5-978c477db-89wrk" Sep 07 16:40:51.366986 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:51.366903 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d7b6a5-978c477db-89wrk" Sep 07 16:40:51.637232 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:51.637049 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d7b6a5-978c477db-89wrk"] Sep 07 16:40:51.639582 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:40:51.639549 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b2e3e0a_f07d_4c05_b56b_46a35a9bd810.slice/crio-bec32494c6eec0070b486842fc787f03c9225d2511e538cfd49e0182c7b7a7ac WatchSource:0}: Error finding container bec32494c6eec0070b486842fc787f03c9225d2511e538cfd49e0182c7b7a7ac: Status 404 returned error can't find the container with id bec32494c6eec0070b486842fc787f03c9225d2511e538cfd49e0182c7b7a7ac Sep 07 16:40:52.374039 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:52.373989 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d7b6a5-978c477db-89wrk" event={"ID":"9b2e3e0a-f07d-4c05-b56b-46a35a9bd810","Type":"ContainerStarted","Data":"8d14bfa9281a7c288061259a237ba2b0e6731663c13458451327ed245bd6f823"} Sep 07 16:40:52.374553 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:52.374053 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d7b6a5-978c477db-89wrk" event={"ID":"9b2e3e0a-f07d-4c05-b56b-46a35a9bd810","Type":"ContainerStarted","Data":"bec32494c6eec0070b486842fc787f03c9225d2511e538cfd49e0182c7b7a7ac"} Sep 07 16:40:52.374553 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:52.374119 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d7b6a5-978c477db-89wrk" Sep 07 16:40:52.392273 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:40:52.392226 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d7b6a5-978c477db-89wrk" podStartSLOduration=1.392210012 podStartE2EDuration="1.392210012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:40:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:40:52.391173467 +0000 UTC m=+1281.426396543" watchObservedRunningTime="2026-09-07 16:40:52.392210012 +0000 UTC m=+1281.427433082" Sep 07 16:41:03.380606 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:03.380575 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d7b6a5-978c477db-89wrk" Sep 07 16:41:11.471108 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:11.471073 2580 generic.go:356] "Generic (PLEG): container finished" podID="ed56d248-38d2-42d3-bb8c-31b9c14f3c2d" containerID="1a8a0b5d9bebc753f087e12f86bb9e8a42af8e12c3bfade71e6125b63d4c9282" exitCode=0 Sep 07 16:41:11.471583 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:11.471144 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4" event={"ID":"ed56d248-38d2-42d3-bb8c-31b9c14f3c2d","Type":"ContainerDied","Data":"1a8a0b5d9bebc753f087e12f86bb9e8a42af8e12c3bfade71e6125b63d4c9282"} Sep 07 16:41:11.471583 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:11.471539 2580 scope.go:117] "RemoveContainer" containerID="1a8a0b5d9bebc753f087e12f86bb9e8a42af8e12c3bfade71e6125b63d4c9282" Sep 07 16:41:11.472690 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:11.472666 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:41:11.472751 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:11.472738 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:41:11.472786 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:11.472772 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:41:12.477800 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:12.477760 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4" event={"ID":"ed56d248-38d2-42d3-bb8c-31b9c14f3c2d","Type":"ContainerStarted","Data":"68c845d4e609f940aae3091e2487df689df6f25746a9cf66da7ad9b62bb192b7"} Sep 07 16:41:13.023715 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:13.023672 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4" Sep 07 16:41:13.024217 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:13.024182 2580 patch_prober.go:28] interesting pod/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Sep 07 16:41:13.024353 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:13.024252 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4" podUID="ed56d248-38d2-42d3-bb8c-31b9c14f3c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Sep 07 16:41:14.024237 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:14.024193 2580 patch_prober.go:28] interesting pod/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Sep 07 16:41:14.024732 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:14.024270 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4" podUID="ed56d248-38d2-42d3-bb8c-31b9c14f3c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Sep 07 16:41:15.023747 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:15.023701 2580 patch_prober.go:28] interesting pod/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Sep 07 16:41:15.024075 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:15.024015 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4" podUID="ed56d248-38d2-42d3-bb8c-31b9c14f3c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Sep 07 16:41:16.024138 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:16.024097 2580 patch_prober.go:28] interesting pod/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Sep 07 16:41:16.024714 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:16.024681 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4" podUID="ed56d248-38d2-42d3-bb8c-31b9c14f3c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Sep 07 16:41:17.024558 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:17.024510 2580 patch_prober.go:28] interesting pod/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Sep 07 16:41:17.025090 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:17.024583 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4" podUID="ed56d248-38d2-42d3-bb8c-31b9c14f3c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Sep 07 16:41:18.024584 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:18.024540 2580 patch_prober.go:28] interesting pod/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Sep 07 16:41:18.024980 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:18.024609 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4" podUID="ed56d248-38d2-42d3-bb8c-31b9c14f3c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Sep 07 16:41:18.354557 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:18.354468 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4" Sep 07 16:41:19.028385 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:19.028359 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4" Sep 07 16:41:19.029385 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:19.029364 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4" Sep 07 16:41:32.572285 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:32.571196 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8ba4d4-7c7b47469d-zw2f7"] Sep 07 16:41:32.572285 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:32.571646 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8ba4d4-7c7b47469d-zw2f7" podUID="0694cdbd-ea07-4980-a668-66bca8cb7b2f" containerName="payload-pre-processing" containerID="cri-o://651676560eb400174f26f9445220c6b03f0f07427793516ed0523f602130fe6d" gracePeriod=30 Sep 07 16:41:32.841119 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:32.841089 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8ba4d4-7c7b47469d-zw2f7" Sep 07 16:41:32.917310 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:32.917273 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwtmw\" (UniqueName: \"kubernetes.io/projected/0694cdbd-ea07-4980-a668-66bca8cb7b2f-kube-api-access-fwtmw\") pod \"0694cdbd-ea07-4980-a668-66bca8cb7b2f\" (UID: \"0694cdbd-ea07-4980-a668-66bca8cb7b2f\") " Sep 07 16:41:32.917310 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:32.917317 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0694cdbd-ea07-4980-a668-66bca8cb7b2f-plugins-config-volume\") pod \"0694cdbd-ea07-4980-a668-66bca8cb7b2f\" (UID: \"0694cdbd-ea07-4980-a668-66bca8cb7b2f\") " Sep 07 16:41:32.917751 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:32.917723 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0694cdbd-ea07-4980-a668-66bca8cb7b2f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0694cdbd-ea07-4980-a668-66bca8cb7b2f" (UID: "0694cdbd-ea07-4980-a668-66bca8cb7b2f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 16:41:32.919614 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:32.919592 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0694cdbd-ea07-4980-a668-66bca8cb7b2f-kube-api-access-fwtmw" (OuterVolumeSpecName: "kube-api-access-fwtmw") pod "0694cdbd-ea07-4980-a668-66bca8cb7b2f" (UID: "0694cdbd-ea07-4980-a668-66bca8cb7b2f"). InnerVolumeSpecName "kube-api-access-fwtmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 16:41:33.018248 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:33.018214 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwtmw\" (UniqueName: \"kubernetes.io/projected/0694cdbd-ea07-4980-a668-66bca8cb7b2f-kube-api-access-fwtmw\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:41:33.018248 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:33.018244 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0694cdbd-ea07-4980-a668-66bca8cb7b2f-plugins-config-volume\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:41:33.582274 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:33.582240 2580 generic.go:356] "Generic (PLEG): container finished" podID="0694cdbd-ea07-4980-a668-66bca8cb7b2f" containerID="651676560eb400174f26f9445220c6b03f0f07427793516ed0523f602130fe6d" exitCode=0 Sep 07 16:41:33.582733 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:33.582292 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8ba4d4-7c7b47469d-zw2f7" Sep 07 16:41:33.582733 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:33.582306 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8ba4d4-7c7b47469d-zw2f7" event={"ID":"0694cdbd-ea07-4980-a668-66bca8cb7b2f","Type":"ContainerDied","Data":"651676560eb400174f26f9445220c6b03f0f07427793516ed0523f602130fe6d"} Sep 07 16:41:33.582733 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:33.582373 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8ba4d4-7c7b47469d-zw2f7" event={"ID":"0694cdbd-ea07-4980-a668-66bca8cb7b2f","Type":"ContainerDied","Data":"8840f0b20c0775d9dbde4780ea6eb0b55b9346d5836f6d7e65e6246f911fe5e7"} Sep 07 16:41:33.582733 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:33.582394 2580 scope.go:117] "RemoveContainer" containerID="651676560eb400174f26f9445220c6b03f0f07427793516ed0523f602130fe6d" Sep 07 16:41:33.591448 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:33.591428 2580 scope.go:117] "RemoveContainer" containerID="651676560eb400174f26f9445220c6b03f0f07427793516ed0523f602130fe6d" Sep 07 16:41:33.591703 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:41:33.591683 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"651676560eb400174f26f9445220c6b03f0f07427793516ed0523f602130fe6d\": container with ID starting with 651676560eb400174f26f9445220c6b03f0f07427793516ed0523f602130fe6d not found: ID does not exist" containerID="651676560eb400174f26f9445220c6b03f0f07427793516ed0523f602130fe6d" Sep 07 16:41:33.591757 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:33.591715 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"651676560eb400174f26f9445220c6b03f0f07427793516ed0523f602130fe6d"} err="failed to get container status \"651676560eb400174f26f9445220c6b03f0f07427793516ed0523f602130fe6d\": rpc error: code = NotFound desc = could not find container \"651676560eb400174f26f9445220c6b03f0f07427793516ed0523f602130fe6d\": container with ID starting with 651676560eb400174f26f9445220c6b03f0f07427793516ed0523f602130fe6d not found: ID does not exist" Sep 07 16:41:33.596843 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:33.596821 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8ba4d4-7c7b47469d-zw2f7"] Sep 07 16:41:33.602410 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:33.602387 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8ba4d4-7c7b47469d-zw2f7"] Sep 07 16:41:35.418981 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:35.418948 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0694cdbd-ea07-4980-a668-66bca8cb7b2f" path="/var/lib/kubelet/pods/0694cdbd-ea07-4980-a668-66bca8cb7b2f/volumes" Sep 07 16:41:39.315501 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:39.315458 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj"] Sep 07 16:41:39.315903 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:39.315694 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj" podUID="515038a5-c027-4894-ac05-08db9c29f3c6" containerName="istio-proxy" containerID="cri-o://f78f38f6dffeb7b521d8015ed227157e1565330417fa8924a99935c6c19be415" gracePeriod=30 Sep 07 16:41:39.649335 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:39.649238 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d7b6a5-h65rg"] Sep 07 16:41:39.649678 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:39.649660 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0694cdbd-ea07-4980-a668-66bca8cb7b2f" containerName="payload-pre-processing" Sep 07 16:41:39.649763 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:39.649682 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0694cdbd-ea07-4980-a668-66bca8cb7b2f" containerName="payload-pre-processing" Sep 07 16:41:39.649814 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:39.649802 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0694cdbd-ea07-4980-a668-66bca8cb7b2f" containerName="payload-pre-processing" Sep 07 16:41:39.652929 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:39.652905 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d7b6a5-h65rg" Sep 07 16:41:39.654912 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:39.654886 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 07 16:41:39.655072 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:39.654891 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jpjxt\"" Sep 07 16:41:39.658942 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:39.658917 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d7b6a5-h65rg"] Sep 07 16:41:39.790738 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:39.790695 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/620c36cf-3ac6-477d-a597-5d27c07473e6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d7b6a5-h65rg\" (UID: \"620c36cf-3ac6-477d-a597-5d27c07473e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d7b6a5-h65rg" Sep 07 16:41:39.891743 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:39.891701 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/620c36cf-3ac6-477d-a597-5d27c07473e6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d7b6a5-h65rg\" (UID: \"620c36cf-3ac6-477d-a597-5d27c07473e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d7b6a5-h65rg" Sep 07 16:41:39.892356 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:39.892335 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/620c36cf-3ac6-477d-a597-5d27c07473e6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d7b6a5-h65rg\" (UID: \"620c36cf-3ac6-477d-a597-5d27c07473e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d7b6a5-h65rg" Sep 07 16:41:39.964507 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:39.964466 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d7b6a5-h65rg" Sep 07 16:41:40.232768 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:40.232713 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d7b6a5-h65rg"] Sep 07 16:41:40.235009 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:41:40.234980 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod620c36cf_3ac6_477d_a597_5d27c07473e6.slice/crio-8f92c662412c13a8b300e3101e4ad572b56c8e2e14058e3122ed21b6d39ec6f8 WatchSource:0}: Error finding container 8f92c662412c13a8b300e3101e4ad572b56c8e2e14058e3122ed21b6d39ec6f8: Status 404 returned error can't find the container with id 8f92c662412c13a8b300e3101e4ad572b56c8e2e14058e3122ed21b6d39ec6f8 Sep 07 16:41:40.618738 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:40.618642 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d7b6a5-h65rg" event={"ID":"620c36cf-3ac6-477d-a597-5d27c07473e6","Type":"ContainerStarted","Data":"6c27ccbe9f2458ad7380109a15bd3256705374f5fcb31fc49a340341a396a38b"} Sep 07 16:41:40.618738 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:40.618704 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d7b6a5-h65rg" event={"ID":"620c36cf-3ac6-477d-a597-5d27c07473e6","Type":"ContainerStarted","Data":"8f92c662412c13a8b300e3101e4ad572b56c8e2e14058e3122ed21b6d39ec6f8"} Sep 07 16:41:40.636256 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:40.636205 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d7b6a5-h65rg" podStartSLOduration=1.636187002 podStartE2EDuration="1.636187002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:41:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:41:40.63468134 +0000 UTC m=+1329.669904411" watchObservedRunningTime="2026-09-07 16:41:40.636187002 +0000 UTC m=+1329.671410071" Sep 07 16:41:41.625317 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:41.625280 2580 generic.go:356] "Generic (PLEG): container finished" podID="620c36cf-3ac6-477d-a597-5d27c07473e6" containerID="6c27ccbe9f2458ad7380109a15bd3256705374f5fcb31fc49a340341a396a38b" exitCode=0 Sep 07 16:41:41.625731 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:41.625368 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d7b6a5-h65rg" event={"ID":"620c36cf-3ac6-477d-a597-5d27c07473e6","Type":"ContainerDied","Data":"6c27ccbe9f2458ad7380109a15bd3256705374f5fcb31fc49a340341a396a38b"} Sep 07 16:41:42.770038 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:42.770000 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d7b6a5-h65rg" Sep 07 16:41:42.921368 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:42.921281 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/620c36cf-3ac6-477d-a597-5d27c07473e6-maas-api-service-ca\") pod \"620c36cf-3ac6-477d-a597-5d27c07473e6\" (UID: \"620c36cf-3ac6-477d-a597-5d27c07473e6\") " Sep 07 16:41:42.921633 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:42.921609 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/620c36cf-3ac6-477d-a597-5d27c07473e6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "620c36cf-3ac6-477d-a597-5d27c07473e6" (UID: "620c36cf-3ac6-477d-a597-5d27c07473e6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 16:41:43.022179 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:43.022145 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/620c36cf-3ac6-477d-a597-5d27c07473e6-maas-api-service-ca\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:41:43.637608 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:43.637577 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d7b6a5-h65rg" Sep 07 16:41:43.637608 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:43.637606 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d7b6a5-h65rg" event={"ID":"620c36cf-3ac6-477d-a597-5d27c07473e6","Type":"ContainerDied","Data":"8f92c662412c13a8b300e3101e4ad572b56c8e2e14058e3122ed21b6d39ec6f8"} Sep 07 16:41:43.637859 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:43.637645 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f92c662412c13a8b300e3101e4ad572b56c8e2e14058e3122ed21b6d39ec6f8" Sep 07 16:41:44.589073 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:44.589000 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj" Sep 07 16:41:44.644422 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:44.644388 2580 generic.go:356] "Generic (PLEG): container finished" podID="515038a5-c027-4894-ac05-08db9c29f3c6" containerID="f78f38f6dffeb7b521d8015ed227157e1565330417fa8924a99935c6c19be415" exitCode=0 Sep 07 16:41:44.644593 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:44.644440 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj" Sep 07 16:41:44.644593 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:44.644468 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj" event={"ID":"515038a5-c027-4894-ac05-08db9c29f3c6","Type":"ContainerDied","Data":"f78f38f6dffeb7b521d8015ed227157e1565330417fa8924a99935c6c19be415"} Sep 07 16:41:44.644593 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:44.644525 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj" event={"ID":"515038a5-c027-4894-ac05-08db9c29f3c6","Type":"ContainerDied","Data":"100f60558bfc07a0e45ee4e3ff155243829dbb0846b6f2eab1f19695529dcae7"} Sep 07 16:41:44.644593 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:44.644577 2580 scope.go:117] "RemoveContainer" containerID="f78f38f6dffeb7b521d8015ed227157e1565330417fa8924a99935c6c19be415" Sep 07 16:41:44.653167 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:44.653145 2580 scope.go:117] "RemoveContainer" containerID="f78b03fa150d33db9a70dd9112a274782f8c7e65c11b06f7faa20e3e2a59cbb0" Sep 07 16:41:44.661047 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:44.661002 2580 scope.go:117] "RemoveContainer" containerID="f78f38f6dffeb7b521d8015ed227157e1565330417fa8924a99935c6c19be415" Sep 07 16:41:44.661318 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:41:44.661299 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f78f38f6dffeb7b521d8015ed227157e1565330417fa8924a99935c6c19be415\": container with ID starting with f78f38f6dffeb7b521d8015ed227157e1565330417fa8924a99935c6c19be415 not found: ID does not exist" containerID="f78f38f6dffeb7b521d8015ed227157e1565330417fa8924a99935c6c19be415" Sep 07 16:41:44.661366 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:44.661330 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f78f38f6dffeb7b521d8015ed227157e1565330417fa8924a99935c6c19be415"} err="failed to get container status \"f78f38f6dffeb7b521d8015ed227157e1565330417fa8924a99935c6c19be415\": rpc error: code = NotFound desc = could not find container \"f78f38f6dffeb7b521d8015ed227157e1565330417fa8924a99935c6c19be415\": container with ID starting with f78f38f6dffeb7b521d8015ed227157e1565330417fa8924a99935c6c19be415 not found: ID does not exist" Sep 07 16:41:44.661366 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:44.661348 2580 scope.go:117] "RemoveContainer" containerID="f78b03fa150d33db9a70dd9112a274782f8c7e65c11b06f7faa20e3e2a59cbb0" Sep 07 16:41:44.661571 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:41:44.661555 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f78b03fa150d33db9a70dd9112a274782f8c7e65c11b06f7faa20e3e2a59cbb0\": container with ID starting with f78b03fa150d33db9a70dd9112a274782f8c7e65c11b06f7faa20e3e2a59cbb0 not found: ID does not exist" containerID="f78b03fa150d33db9a70dd9112a274782f8c7e65c11b06f7faa20e3e2a59cbb0" Sep 07 16:41:44.661629 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:44.661580 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f78b03fa150d33db9a70dd9112a274782f8c7e65c11b06f7faa20e3e2a59cbb0"} err="failed to get container status \"f78b03fa150d33db9a70dd9112a274782f8c7e65c11b06f7faa20e3e2a59cbb0\": rpc error: code = NotFound desc = could not find container \"f78b03fa150d33db9a70dd9112a274782f8c7e65c11b06f7faa20e3e2a59cbb0\": container with ID starting with f78b03fa150d33db9a70dd9112a274782f8c7e65c11b06f7faa20e3e2a59cbb0 not found: ID does not exist" Sep 07 16:41:44.740572 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:44.740543 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/515038a5-c027-4894-ac05-08db9c29f3c6-workload-certs\") pod \"515038a5-c027-4894-ac05-08db9c29f3c6\" (UID: \"515038a5-c027-4894-ac05-08db9c29f3c6\") " Sep 07 16:41:44.740736 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:44.740584 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/515038a5-c027-4894-ac05-08db9c29f3c6-istio-data\") pod \"515038a5-c027-4894-ac05-08db9c29f3c6\" (UID: \"515038a5-c027-4894-ac05-08db9c29f3c6\") " Sep 07 16:41:44.740736 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:44.740611 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/515038a5-c027-4894-ac05-08db9c29f3c6-istio-podinfo\") pod \"515038a5-c027-4894-ac05-08db9c29f3c6\" (UID: \"515038a5-c027-4894-ac05-08db9c29f3c6\") " Sep 07 16:41:44.740736 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:44.740637 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/515038a5-c027-4894-ac05-08db9c29f3c6-workload-socket\") pod \"515038a5-c027-4894-ac05-08db9c29f3c6\" (UID: \"515038a5-c027-4894-ac05-08db9c29f3c6\") " Sep 07 16:41:44.740736 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:44.740652 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/515038a5-c027-4894-ac05-08db9c29f3c6-istio-token\") pod \"515038a5-c027-4894-ac05-08db9c29f3c6\" (UID: \"515038a5-c027-4894-ac05-08db9c29f3c6\") " Sep 07 16:41:44.740736 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:44.740680 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75m42\" (UniqueName: \"kubernetes.io/projected/515038a5-c027-4894-ac05-08db9c29f3c6-kube-api-access-75m42\") pod \"515038a5-c027-4894-ac05-08db9c29f3c6\" (UID: \"515038a5-c027-4894-ac05-08db9c29f3c6\") " Sep 07 16:41:44.740996 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:44.740744 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/515038a5-c027-4894-ac05-08db9c29f3c6-credential-socket\") pod \"515038a5-c027-4894-ac05-08db9c29f3c6\" (UID: \"515038a5-c027-4894-ac05-08db9c29f3c6\") " Sep 07 16:41:44.740996 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:44.740783 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/515038a5-c027-4894-ac05-08db9c29f3c6-istio-envoy\") pod \"515038a5-c027-4894-ac05-08db9c29f3c6\" (UID: \"515038a5-c027-4894-ac05-08db9c29f3c6\") " Sep 07 16:41:44.740996 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:44.740810 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/515038a5-c027-4894-ac05-08db9c29f3c6-istiod-ca-cert\") pod \"515038a5-c027-4894-ac05-08db9c29f3c6\" (UID: \"515038a5-c027-4894-ac05-08db9c29f3c6\") " Sep 07 16:41:44.740996 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:44.740877 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/515038a5-c027-4894-ac05-08db9c29f3c6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "515038a5-c027-4894-ac05-08db9c29f3c6" (UID: "515038a5-c027-4894-ac05-08db9c29f3c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 16:41:44.740996 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:44.740886 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/515038a5-c027-4894-ac05-08db9c29f3c6-istio-data" (OuterVolumeSpecName: "istio-data") pod "515038a5-c027-4894-ac05-08db9c29f3c6" (UID: "515038a5-c027-4894-ac05-08db9c29f3c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 16:41:44.740996 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:44.740975 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/515038a5-c027-4894-ac05-08db9c29f3c6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "515038a5-c027-4894-ac05-08db9c29f3c6" (UID: "515038a5-c027-4894-ac05-08db9c29f3c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 16:41:44.741339 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:44.741199 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/515038a5-c027-4894-ac05-08db9c29f3c6-istio-data\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:41:44.741339 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:44.741219 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/515038a5-c027-4894-ac05-08db9c29f3c6-workload-socket\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:41:44.741339 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:44.741233 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/515038a5-c027-4894-ac05-08db9c29f3c6-workload-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:41:44.741339 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:44.741288 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/515038a5-c027-4894-ac05-08db9c29f3c6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "515038a5-c027-4894-ac05-08db9c29f3c6" (UID: "515038a5-c027-4894-ac05-08db9c29f3c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 16:41:44.741339 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:44.741321 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/515038a5-c027-4894-ac05-08db9c29f3c6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "515038a5-c027-4894-ac05-08db9c29f3c6" (UID: "515038a5-c027-4894-ac05-08db9c29f3c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 16:41:44.743133 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:44.743106 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/515038a5-c027-4894-ac05-08db9c29f3c6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "515038a5-c027-4894-ac05-08db9c29f3c6" (UID: "515038a5-c027-4894-ac05-08db9c29f3c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 16:41:44.743254 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:44.743181 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/515038a5-c027-4894-ac05-08db9c29f3c6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "515038a5-c027-4894-ac05-08db9c29f3c6" (UID: "515038a5-c027-4894-ac05-08db9c29f3c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 07 16:41:44.743321 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:44.743276 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/515038a5-c027-4894-ac05-08db9c29f3c6-kube-api-access-75m42" (OuterVolumeSpecName: "kube-api-access-75m42") pod "515038a5-c027-4894-ac05-08db9c29f3c6" (UID: "515038a5-c027-4894-ac05-08db9c29f3c6"). InnerVolumeSpecName "kube-api-access-75m42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 16:41:44.743321 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:44.743292 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/515038a5-c027-4894-ac05-08db9c29f3c6-istio-token" (OuterVolumeSpecName: "istio-token") pod "515038a5-c027-4894-ac05-08db9c29f3c6" (UID: "515038a5-c027-4894-ac05-08db9c29f3c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 16:41:44.842455 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:44.842376 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75m42\" (UniqueName: \"kubernetes.io/projected/515038a5-c027-4894-ac05-08db9c29f3c6-kube-api-access-75m42\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:41:44.842455 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:44.842403 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/515038a5-c027-4894-ac05-08db9c29f3c6-credential-socket\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:41:44.842455 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:44.842413 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/515038a5-c027-4894-ac05-08db9c29f3c6-istio-envoy\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:41:44.842455 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:44.842422 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/515038a5-c027-4894-ac05-08db9c29f3c6-istiod-ca-cert\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:41:44.842455 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:44.842431 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/515038a5-c027-4894-ac05-08db9c29f3c6-istio-podinfo\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:41:44.842455 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:44.842439 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/515038a5-c027-4894-ac05-08db9c29f3c6-istio-token\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:41:44.966778 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:44.966753 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj"] Sep 07 16:41:44.969249 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:44.969225 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-8ba4d4-openshift-default-545cc9b9cd-vlfgj"] Sep 07 16:41:45.425200 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:45.425155 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="515038a5-c027-4894-ac05-08db9c29f3c6" path="/var/lib/kubelet/pods/515038a5-c027-4894-ac05-08db9c29f3c6/volumes" Sep 07 16:41:55.229099 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:55.228996 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d7b6a5-978c477db-89wrk"] Sep 07 16:41:55.229521 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:55.229281 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d7b6a5-978c477db-89wrk" podUID="9b2e3e0a-f07d-4c05-b56b-46a35a9bd810" containerName="payload-pre-processing" containerID="cri-o://8d14bfa9281a7c288061259a237ba2b0e6731663c13458451327ed245bd6f823" gracePeriod=30 Sep 07 16:41:55.370038 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:55.369989 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d7b6a5-h65rg"] Sep 07 16:41:55.373787 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:55.373757 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d7b6a5-h65rg"] Sep 07 16:41:55.420197 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:55.420170 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="620c36cf-3ac6-477d-a597-5d27c07473e6" path="/var/lib/kubelet/pods/620c36cf-3ac6-477d-a597-5d27c07473e6/volumes" Sep 07 16:41:55.492062 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:55.491981 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d7b6a5-978c477db-89wrk" Sep 07 16:41:55.541890 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:55.541854 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b2e3e0a-f07d-4c05-b56b-46a35a9bd810-plugins-config-volume\") pod \"9b2e3e0a-f07d-4c05-b56b-46a35a9bd810\" (UID: \"9b2e3e0a-f07d-4c05-b56b-46a35a9bd810\") " Sep 07 16:41:55.542094 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:55.541913 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qgvl5\" (UniqueName: \"kubernetes.io/projected/9b2e3e0a-f07d-4c05-b56b-46a35a9bd810-kube-api-access-qgvl5\") pod \"9b2e3e0a-f07d-4c05-b56b-46a35a9bd810\" (UID: \"9b2e3e0a-f07d-4c05-b56b-46a35a9bd810\") " Sep 07 16:41:55.542269 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:55.542241 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b2e3e0a-f07d-4c05-b56b-46a35a9bd810-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9b2e3e0a-f07d-4c05-b56b-46a35a9bd810" (UID: "9b2e3e0a-f07d-4c05-b56b-46a35a9bd810"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 16:41:55.544347 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:55.544325 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b2e3e0a-f07d-4c05-b56b-46a35a9bd810-kube-api-access-qgvl5" (OuterVolumeSpecName: "kube-api-access-qgvl5") pod "9b2e3e0a-f07d-4c05-b56b-46a35a9bd810" (UID: "9b2e3e0a-f07d-4c05-b56b-46a35a9bd810"). InnerVolumeSpecName "kube-api-access-qgvl5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 16:41:55.642958 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:55.642910 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b2e3e0a-f07d-4c05-b56b-46a35a9bd810-plugins-config-volume\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:41:55.642958 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:55.642952 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgvl5\" (UniqueName: \"kubernetes.io/projected/9b2e3e0a-f07d-4c05-b56b-46a35a9bd810-kube-api-access-qgvl5\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:41:55.700309 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:55.700273 2580 generic.go:356] "Generic (PLEG): container finished" podID="9b2e3e0a-f07d-4c05-b56b-46a35a9bd810" containerID="8d14bfa9281a7c288061259a237ba2b0e6731663c13458451327ed245bd6f823" exitCode=0 Sep 07 16:41:55.700512 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:55.700325 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d7b6a5-978c477db-89wrk" Sep 07 16:41:55.700512 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:55.700327 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d7b6a5-978c477db-89wrk" event={"ID":"9b2e3e0a-f07d-4c05-b56b-46a35a9bd810","Type":"ContainerDied","Data":"8d14bfa9281a7c288061259a237ba2b0e6731663c13458451327ed245bd6f823"} Sep 07 16:41:55.700512 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:55.700442 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d7b6a5-978c477db-89wrk" event={"ID":"9b2e3e0a-f07d-4c05-b56b-46a35a9bd810","Type":"ContainerDied","Data":"bec32494c6eec0070b486842fc787f03c9225d2511e538cfd49e0182c7b7a7ac"} Sep 07 16:41:55.700512 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:55.700462 2580 scope.go:117] "RemoveContainer" containerID="8d14bfa9281a7c288061259a237ba2b0e6731663c13458451327ed245bd6f823" Sep 07 16:41:55.709592 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:55.709572 2580 scope.go:117] "RemoveContainer" containerID="8d14bfa9281a7c288061259a237ba2b0e6731663c13458451327ed245bd6f823" Sep 07 16:41:55.709900 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:41:55.709883 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d14bfa9281a7c288061259a237ba2b0e6731663c13458451327ed245bd6f823\": container with ID starting with 8d14bfa9281a7c288061259a237ba2b0e6731663c13458451327ed245bd6f823 not found: ID does not exist" containerID="8d14bfa9281a7c288061259a237ba2b0e6731663c13458451327ed245bd6f823" Sep 07 16:41:55.709944 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:55.709910 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d14bfa9281a7c288061259a237ba2b0e6731663c13458451327ed245bd6f823"} err="failed to get container status \"8d14bfa9281a7c288061259a237ba2b0e6731663c13458451327ed245bd6f823\": rpc error: code = NotFound desc = could not find container \"8d14bfa9281a7c288061259a237ba2b0e6731663c13458451327ed245bd6f823\": container with ID starting with 8d14bfa9281a7c288061259a237ba2b0e6731663c13458451327ed245bd6f823 not found: ID does not exist" Sep 07 16:41:55.721287 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:55.721256 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d7b6a5-978c477db-89wrk"] Sep 07 16:41:55.724312 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:55.724274 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d7b6a5-978c477db-89wrk"] Sep 07 16:41:57.418005 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:41:57.417971 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b2e3e0a-f07d-4c05-b56b-46a35a9bd810" path="/var/lib/kubelet/pods/9b2e3e0a-f07d-4c05-b56b-46a35a9bd810/volumes" Sep 07 16:42:01.581185 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:01.581153 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4"] Sep 07 16:42:01.581643 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:01.581387 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4" podUID="ed56d248-38d2-42d3-bb8c-31b9c14f3c2d" containerName="istio-proxy" containerID="cri-o://68c845d4e609f940aae3091e2487df689df6f25746a9cf66da7ad9b62bb192b7" gracePeriod=30 Sep 07 16:42:01.955501 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:01.955460 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bd6c5a-x8jtw"] Sep 07 16:42:01.955833 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:01.955820 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="620c36cf-3ac6-477d-a597-5d27c07473e6" containerName="revoke-keys" Sep 07 16:42:01.955878 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:01.955835 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="620c36cf-3ac6-477d-a597-5d27c07473e6" containerName="revoke-keys" Sep 07 16:42:01.955878 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:01.955848 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="515038a5-c027-4894-ac05-08db9c29f3c6" containerName="istio-proxy" Sep 07 16:42:01.955878 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:01.955854 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="515038a5-c027-4894-ac05-08db9c29f3c6" containerName="istio-proxy" Sep 07 16:42:01.955878 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:01.955863 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="515038a5-c027-4894-ac05-08db9c29f3c6" containerName="istio-proxy" Sep 07 16:42:01.955878 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:01.955868 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="515038a5-c027-4894-ac05-08db9c29f3c6" containerName="istio-proxy" Sep 07 16:42:01.955878 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:01.955873 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b2e3e0a-f07d-4c05-b56b-46a35a9bd810" containerName="payload-pre-processing" Sep 07 16:42:01.955878 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:01.955878 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b2e3e0a-f07d-4c05-b56b-46a35a9bd810" containerName="payload-pre-processing" Sep 07 16:42:01.956158 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:01.955934 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="515038a5-c027-4894-ac05-08db9c29f3c6" containerName="istio-proxy" Sep 07 16:42:01.956158 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:01.955941 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b2e3e0a-f07d-4c05-b56b-46a35a9bd810" containerName="payload-pre-processing" Sep 07 16:42:01.956158 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:01.955947 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="620c36cf-3ac6-477d-a597-5d27c07473e6" containerName="revoke-keys" Sep 07 16:42:01.958850 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:01.958831 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bd6c5a-x8jtw" Sep 07 16:42:01.961133 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:01.961109 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 07 16:42:01.961251 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:01.961215 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jpjxt\"" Sep 07 16:42:01.966723 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:01.966696 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bd6c5a-x8jtw"] Sep 07 16:42:02.004377 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:02.004344 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80a1687e-7bb9-4da4-8266-da9419cb6f4f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-bd6c5a-x8jtw\" (UID: \"80a1687e-7bb9-4da4-8266-da9419cb6f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bd6c5a-x8jtw" Sep 07 16:42:02.105575 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:02.105538 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80a1687e-7bb9-4da4-8266-da9419cb6f4f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-bd6c5a-x8jtw\" (UID: \"80a1687e-7bb9-4da4-8266-da9419cb6f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bd6c5a-x8jtw" Sep 07 16:42:02.106174 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:02.106155 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80a1687e-7bb9-4da4-8266-da9419cb6f4f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-bd6c5a-x8jtw\" (UID: \"80a1687e-7bb9-4da4-8266-da9419cb6f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bd6c5a-x8jtw" Sep 07 16:42:02.270544 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:02.270452 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bd6c5a-x8jtw" Sep 07 16:42:02.539492 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:02.539410 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bd6c5a-x8jtw"] Sep 07 16:42:02.543135 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:42:02.543103 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80a1687e_7bb9_4da4_8266_da9419cb6f4f.slice/crio-cdb067f83ec894de32745982e8784952b1902ca885186f99cffe5b74c1f8f3f1 WatchSource:0}: Error finding container cdb067f83ec894de32745982e8784952b1902ca885186f99cffe5b74c1f8f3f1: Status 404 returned error can't find the container with id cdb067f83ec894de32745982e8784952b1902ca885186f99cffe5b74c1f8f3f1 Sep 07 16:42:02.732920 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:02.732882 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bd6c5a-x8jtw" event={"ID":"80a1687e-7bb9-4da4-8266-da9419cb6f4f","Type":"ContainerStarted","Data":"cdb067f83ec894de32745982e8784952b1902ca885186f99cffe5b74c1f8f3f1"} Sep 07 16:42:03.739658 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:03.739624 2580 generic.go:356] "Generic (PLEG): container finished" podID="80a1687e-7bb9-4da4-8266-da9419cb6f4f" containerID="fdc043b2048faaa72363fd5c600fd134bb33e03d32d7207d527410b2c752cb31" exitCode=0 Sep 07 16:42:03.740124 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:03.739674 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bd6c5a-x8jtw" event={"ID":"80a1687e-7bb9-4da4-8266-da9419cb6f4f","Type":"ContainerDied","Data":"fdc043b2048faaa72363fd5c600fd134bb33e03d32d7207d527410b2c752cb31"} Sep 07 16:42:04.031327 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:04.031238 2580 patch_prober.go:28] interesting pod/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 07 16:42:04.031327 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:04.031294 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4" podUID="ed56d248-38d2-42d3-bb8c-31b9c14f3c2d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 07 16:42:04.883193 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:04.883167 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bd6c5a-x8jtw" Sep 07 16:42:04.930622 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:04.930590 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80a1687e-7bb9-4da4-8266-da9419cb6f4f-maas-api-service-ca\") pod \"80a1687e-7bb9-4da4-8266-da9419cb6f4f\" (UID: \"80a1687e-7bb9-4da4-8266-da9419cb6f4f\") " Sep 07 16:42:04.930963 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:04.930943 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80a1687e-7bb9-4da4-8266-da9419cb6f4f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "80a1687e-7bb9-4da4-8266-da9419cb6f4f" (UID: "80a1687e-7bb9-4da4-8266-da9419cb6f4f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 16:42:05.032152 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:05.032070 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80a1687e-7bb9-4da4-8266-da9419cb6f4f-maas-api-service-ca\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:42:05.753462 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:05.753433 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bd6c5a-x8jtw" Sep 07 16:42:05.753637 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:05.753430 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bd6c5a-x8jtw" event={"ID":"80a1687e-7bb9-4da4-8266-da9419cb6f4f","Type":"ContainerDied","Data":"cdb067f83ec894de32745982e8784952b1902ca885186f99cffe5b74c1f8f3f1"} Sep 07 16:42:05.753637 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:05.753555 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cdb067f83ec894de32745982e8784952b1902ca885186f99cffe5b74c1f8f3f1" Sep 07 16:42:06.761331 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:06.761299 2580 generic.go:356] "Generic (PLEG): container finished" podID="ed56d248-38d2-42d3-bb8c-31b9c14f3c2d" containerID="68c845d4e609f940aae3091e2487df689df6f25746a9cf66da7ad9b62bb192b7" exitCode=0 Sep 07 16:42:06.761776 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:06.761347 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4" event={"ID":"ed56d248-38d2-42d3-bb8c-31b9c14f3c2d","Type":"ContainerDied","Data":"68c845d4e609f940aae3091e2487df689df6f25746a9cf66da7ad9b62bb192b7"} Sep 07 16:42:06.761776 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:06.761401 2580 scope.go:117] "RemoveContainer" containerID="1a8a0b5d9bebc753f087e12f86bb9e8a42af8e12c3bfade71e6125b63d4c9282" Sep 07 16:42:06.836631 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:06.836606 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4" Sep 07 16:42:06.952733 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:06.952701 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed56d248-38d2-42d3-bb8c-31b9c14f3c2d-istio-envoy\") pod \"ed56d248-38d2-42d3-bb8c-31b9c14f3c2d\" (UID: \"ed56d248-38d2-42d3-bb8c-31b9c14f3c2d\") " Sep 07 16:42:06.952880 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:06.952742 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5l7x\" (UniqueName: \"kubernetes.io/projected/ed56d248-38d2-42d3-bb8c-31b9c14f3c2d-kube-api-access-m5l7x\") pod \"ed56d248-38d2-42d3-bb8c-31b9c14f3c2d\" (UID: \"ed56d248-38d2-42d3-bb8c-31b9c14f3c2d\") " Sep 07 16:42:06.952880 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:06.952775 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed56d248-38d2-42d3-bb8c-31b9c14f3c2d-istio-data\") pod \"ed56d248-38d2-42d3-bb8c-31b9c14f3c2d\" (UID: \"ed56d248-38d2-42d3-bb8c-31b9c14f3c2d\") " Sep 07 16:42:06.952880 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:06.952806 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed56d248-38d2-42d3-bb8c-31b9c14f3c2d-istiod-ca-cert\") pod \"ed56d248-38d2-42d3-bb8c-31b9c14f3c2d\" (UID: \"ed56d248-38d2-42d3-bb8c-31b9c14f3c2d\") " Sep 07 16:42:06.952880 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:06.952832 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed56d248-38d2-42d3-bb8c-31b9c14f3c2d-istio-token\") pod \"ed56d248-38d2-42d3-bb8c-31b9c14f3c2d\" (UID: \"ed56d248-38d2-42d3-bb8c-31b9c14f3c2d\") " Sep 07 16:42:06.952880 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:06.952850 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed56d248-38d2-42d3-bb8c-31b9c14f3c2d-istio-podinfo\") pod \"ed56d248-38d2-42d3-bb8c-31b9c14f3c2d\" (UID: \"ed56d248-38d2-42d3-bb8c-31b9c14f3c2d\") " Sep 07 16:42:06.952880 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:06.952869 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed56d248-38d2-42d3-bb8c-31b9c14f3c2d-workload-socket\") pod \"ed56d248-38d2-42d3-bb8c-31b9c14f3c2d\" (UID: \"ed56d248-38d2-42d3-bb8c-31b9c14f3c2d\") " Sep 07 16:42:06.953230 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:06.952941 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed56d248-38d2-42d3-bb8c-31b9c14f3c2d-workload-certs\") pod \"ed56d248-38d2-42d3-bb8c-31b9c14f3c2d\" (UID: \"ed56d248-38d2-42d3-bb8c-31b9c14f3c2d\") " Sep 07 16:42:06.953230 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:06.952977 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed56d248-38d2-42d3-bb8c-31b9c14f3c2d-credential-socket\") pod \"ed56d248-38d2-42d3-bb8c-31b9c14f3c2d\" (UID: \"ed56d248-38d2-42d3-bb8c-31b9c14f3c2d\") " Sep 07 16:42:06.953230 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:06.953193 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed56d248-38d2-42d3-bb8c-31b9c14f3c2d-istio-data" (OuterVolumeSpecName: "istio-data") pod "ed56d248-38d2-42d3-bb8c-31b9c14f3c2d" (UID: "ed56d248-38d2-42d3-bb8c-31b9c14f3c2d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 16:42:06.953387 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:06.953295 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed56d248-38d2-42d3-bb8c-31b9c14f3c2d-istio-data\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:42:06.953387 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:06.953295 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed56d248-38d2-42d3-bb8c-31b9c14f3c2d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ed56d248-38d2-42d3-bb8c-31b9c14f3c2d" (UID: "ed56d248-38d2-42d3-bb8c-31b9c14f3c2d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 16:42:06.953387 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:06.953327 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed56d248-38d2-42d3-bb8c-31b9c14f3c2d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ed56d248-38d2-42d3-bb8c-31b9c14f3c2d" (UID: "ed56d248-38d2-42d3-bb8c-31b9c14f3c2d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 16:42:06.953543 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:06.953436 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed56d248-38d2-42d3-bb8c-31b9c14f3c2d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ed56d248-38d2-42d3-bb8c-31b9c14f3c2d" (UID: "ed56d248-38d2-42d3-bb8c-31b9c14f3c2d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 16:42:06.953727 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:06.953705 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed56d248-38d2-42d3-bb8c-31b9c14f3c2d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ed56d248-38d2-42d3-bb8c-31b9c14f3c2d" (UID: "ed56d248-38d2-42d3-bb8c-31b9c14f3c2d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 16:42:06.955273 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:06.955252 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed56d248-38d2-42d3-bb8c-31b9c14f3c2d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ed56d248-38d2-42d3-bb8c-31b9c14f3c2d" (UID: "ed56d248-38d2-42d3-bb8c-31b9c14f3c2d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 16:42:06.955388 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:06.955275 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed56d248-38d2-42d3-bb8c-31b9c14f3c2d-istio-token" (OuterVolumeSpecName: "istio-token") pod "ed56d248-38d2-42d3-bb8c-31b9c14f3c2d" (UID: "ed56d248-38d2-42d3-bb8c-31b9c14f3c2d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 16:42:06.955666 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:06.955648 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed56d248-38d2-42d3-bb8c-31b9c14f3c2d-kube-api-access-m5l7x" (OuterVolumeSpecName: "kube-api-access-m5l7x") pod "ed56d248-38d2-42d3-bb8c-31b9c14f3c2d" (UID: "ed56d248-38d2-42d3-bb8c-31b9c14f3c2d"). InnerVolumeSpecName "kube-api-access-m5l7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 16:42:06.955726 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:06.955678 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed56d248-38d2-42d3-bb8c-31b9c14f3c2d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ed56d248-38d2-42d3-bb8c-31b9c14f3c2d" (UID: "ed56d248-38d2-42d3-bb8c-31b9c14f3c2d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 07 16:42:07.054002 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:07.053968 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed56d248-38d2-42d3-bb8c-31b9c14f3c2d-istiod-ca-cert\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:42:07.054002 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:07.053994 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed56d248-38d2-42d3-bb8c-31b9c14f3c2d-istio-token\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:42:07.054002 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:07.054004 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed56d248-38d2-42d3-bb8c-31b9c14f3c2d-istio-podinfo\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:42:07.054002 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:07.054012 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed56d248-38d2-42d3-bb8c-31b9c14f3c2d-workload-socket\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:42:07.054294 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:07.054034 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed56d248-38d2-42d3-bb8c-31b9c14f3c2d-workload-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:42:07.054294 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:07.054043 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed56d248-38d2-42d3-bb8c-31b9c14f3c2d-credential-socket\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:42:07.054294 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:07.054051 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed56d248-38d2-42d3-bb8c-31b9c14f3c2d-istio-envoy\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:42:07.054294 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:07.054061 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5l7x\" (UniqueName: \"kubernetes.io/projected/ed56d248-38d2-42d3-bb8c-31b9c14f3c2d-kube-api-access-m5l7x\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:42:07.766988 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:07.766945 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4" event={"ID":"ed56d248-38d2-42d3-bb8c-31b9c14f3c2d","Type":"ContainerDied","Data":"f659488c0cfc829204632b66b95e64a12cafcf2d887412ae138ec1f6148043b0"} Sep 07 16:42:07.767457 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:07.766958 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4" Sep 07 16:42:07.767457 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:07.767060 2580 scope.go:117] "RemoveContainer" containerID="68c845d4e609f940aae3091e2487df689df6f25746a9cf66da7ad9b62bb192b7" Sep 07 16:42:07.782254 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:07.782229 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4"] Sep 07 16:42:07.788606 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:07.788582 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-d7b6a5-openshift-default-6d4494749b-cmts4"] Sep 07 16:42:09.418838 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:09.418801 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed56d248-38d2-42d3-bb8c-31b9c14f3c2d" path="/var/lib/kubelet/pods/ed56d248-38d2-42d3-bb8c-31b9c14f3c2d/volumes" Sep 07 16:42:12.478076 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:12.478016 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-bd6c5a-c65f7fb55-smjqr"] Sep 07 16:42:12.478600 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:12.478303 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bd6c5a-c65f7fb55-smjqr" podUID="e6474d96-9621-43cd-bfd9-98079628793b" containerName="payload-processing" containerID="cri-o://bd938ebb3aaaa7eacd5dee4a6eee812b3df07c1d91a708c3b8fec079c39c22b2" gracePeriod=30 Sep 07 16:42:12.647908 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:12.647848 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bd6c5a-x8jtw"] Sep 07 16:42:12.650295 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:12.650266 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bd6c5a-x8jtw"] Sep 07 16:42:12.744326 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:12.744303 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bd6c5a-c65f7fb55-smjqr" Sep 07 16:42:12.793431 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:12.793400 2580 generic.go:356] "Generic (PLEG): container finished" podID="e6474d96-9621-43cd-bfd9-98079628793b" containerID="bd938ebb3aaaa7eacd5dee4a6eee812b3df07c1d91a708c3b8fec079c39c22b2" exitCode=0 Sep 07 16:42:12.793612 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:12.793450 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bd6c5a-c65f7fb55-smjqr" Sep 07 16:42:12.793612 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:12.793461 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bd6c5a-c65f7fb55-smjqr" event={"ID":"e6474d96-9621-43cd-bfd9-98079628793b","Type":"ContainerDied","Data":"bd938ebb3aaaa7eacd5dee4a6eee812b3df07c1d91a708c3b8fec079c39c22b2"} Sep 07 16:42:12.793612 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:12.793500 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bd6c5a-c65f7fb55-smjqr" event={"ID":"e6474d96-9621-43cd-bfd9-98079628793b","Type":"ContainerDied","Data":"124c1160ff7810f3ef416d7eff6c701e474d09359b0ff9c7ed4b2074789e0e59"} Sep 07 16:42:12.793612 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:12.793520 2580 scope.go:117] "RemoveContainer" containerID="bd938ebb3aaaa7eacd5dee4a6eee812b3df07c1d91a708c3b8fec079c39c22b2" Sep 07 16:42:12.801737 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:12.801720 2580 scope.go:117] "RemoveContainer" containerID="bd938ebb3aaaa7eacd5dee4a6eee812b3df07c1d91a708c3b8fec079c39c22b2" Sep 07 16:42:12.802051 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:42:12.802033 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd938ebb3aaaa7eacd5dee4a6eee812b3df07c1d91a708c3b8fec079c39c22b2\": container with ID starting with bd938ebb3aaaa7eacd5dee4a6eee812b3df07c1d91a708c3b8fec079c39c22b2 not found: ID does not exist" containerID="bd938ebb3aaaa7eacd5dee4a6eee812b3df07c1d91a708c3b8fec079c39c22b2" Sep 07 16:42:12.802109 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:12.802062 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd938ebb3aaaa7eacd5dee4a6eee812b3df07c1d91a708c3b8fec079c39c22b2"} err="failed to get container status \"bd938ebb3aaaa7eacd5dee4a6eee812b3df07c1d91a708c3b8fec079c39c22b2\": rpc error: code = NotFound desc = could not find container \"bd938ebb3aaaa7eacd5dee4a6eee812b3df07c1d91a708c3b8fec079c39c22b2\": container with ID starting with bd938ebb3aaaa7eacd5dee4a6eee812b3df07c1d91a708c3b8fec079c39c22b2 not found: ID does not exist" Sep 07 16:42:12.808401 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:12.808375 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6474d96-9621-43cd-bfd9-98079628793b-plugins-config-volume\") pod \"e6474d96-9621-43cd-bfd9-98079628793b\" (UID: \"e6474d96-9621-43cd-bfd9-98079628793b\") " Sep 07 16:42:12.808541 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:12.808519 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fgbbd\" (UniqueName: \"kubernetes.io/projected/e6474d96-9621-43cd-bfd9-98079628793b-kube-api-access-fgbbd\") pod \"e6474d96-9621-43cd-bfd9-98079628793b\" (UID: \"e6474d96-9621-43cd-bfd9-98079628793b\") " Sep 07 16:42:12.808901 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:12.808872 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6474d96-9621-43cd-bfd9-98079628793b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e6474d96-9621-43cd-bfd9-98079628793b" (UID: "e6474d96-9621-43cd-bfd9-98079628793b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 16:42:12.813762 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:12.813340 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6474d96-9621-43cd-bfd9-98079628793b-kube-api-access-fgbbd" (OuterVolumeSpecName: "kube-api-access-fgbbd") pod "e6474d96-9621-43cd-bfd9-98079628793b" (UID: "e6474d96-9621-43cd-bfd9-98079628793b"). InnerVolumeSpecName "kube-api-access-fgbbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 16:42:12.909834 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:12.909801 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgbbd\" (UniqueName: \"kubernetes.io/projected/e6474d96-9621-43cd-bfd9-98079628793b-kube-api-access-fgbbd\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:42:12.909834 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:12.909830 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6474d96-9621-43cd-bfd9-98079628793b-plugins-config-volume\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:42:13.116337 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:13.116308 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-bd6c5a-c65f7fb55-smjqr"] Sep 07 16:42:13.118854 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:13.118830 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-bd6c5a-c65f7fb55-smjqr"] Sep 07 16:42:13.418153 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:13.418068 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80a1687e-7bb9-4da4-8266-da9419cb6f4f" path="/var/lib/kubelet/pods/80a1687e-7bb9-4da4-8266-da9419cb6f4f/volumes" Sep 07 16:42:13.418503 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:13.418488 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6474d96-9621-43cd-bfd9-98079628793b" path="/var/lib/kubelet/pods/e6474d96-9621-43cd-bfd9-98079628793b/volumes" Sep 07 16:42:19.321663 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:19.321630 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4a9a21-2fszs"] Sep 07 16:42:19.322050 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:19.321940 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed56d248-38d2-42d3-bb8c-31b9c14f3c2d" containerName="istio-proxy" Sep 07 16:42:19.322050 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:19.321950 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed56d248-38d2-42d3-bb8c-31b9c14f3c2d" containerName="istio-proxy" Sep 07 16:42:19.322050 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:19.321959 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6474d96-9621-43cd-bfd9-98079628793b" containerName="payload-processing" Sep 07 16:42:19.322050 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:19.321965 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6474d96-9621-43cd-bfd9-98079628793b" containerName="payload-processing" Sep 07 16:42:19.322050 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:19.321973 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80a1687e-7bb9-4da4-8266-da9419cb6f4f" containerName="revoke-keys" Sep 07 16:42:19.322050 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:19.321978 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="80a1687e-7bb9-4da4-8266-da9419cb6f4f" containerName="revoke-keys" Sep 07 16:42:19.322242 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:19.322065 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="80a1687e-7bb9-4da4-8266-da9419cb6f4f" containerName="revoke-keys" Sep 07 16:42:19.322242 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:19.322073 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed56d248-38d2-42d3-bb8c-31b9c14f3c2d" containerName="istio-proxy" Sep 07 16:42:19.322242 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:19.322079 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="515038a5-c027-4894-ac05-08db9c29f3c6" containerName="istio-proxy" Sep 07 16:42:19.322242 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:19.322085 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed56d248-38d2-42d3-bb8c-31b9c14f3c2d" containerName="istio-proxy" Sep 07 16:42:19.322242 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:19.322090 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6474d96-9621-43cd-bfd9-98079628793b" containerName="payload-processing" Sep 07 16:42:19.326082 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:19.326065 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4a9a21-2fszs" Sep 07 16:42:19.328516 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:19.328489 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 07 16:42:19.328638 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:19.328489 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jpjxt\"" Sep 07 16:42:19.331732 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:19.331708 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4a9a21-2fszs"] Sep 07 16:42:19.474035 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:19.473998 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32fa202a-5156-4d4b-bcd6-7d6aefd29448-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4a9a21-2fszs\" (UID: \"32fa202a-5156-4d4b-bcd6-7d6aefd29448\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4a9a21-2fszs" Sep 07 16:42:19.575407 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:19.575310 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32fa202a-5156-4d4b-bcd6-7d6aefd29448-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4a9a21-2fszs\" (UID: \"32fa202a-5156-4d4b-bcd6-7d6aefd29448\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4a9a21-2fszs" Sep 07 16:42:19.575911 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:19.575888 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32fa202a-5156-4d4b-bcd6-7d6aefd29448-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4a9a21-2fszs\" (UID: \"32fa202a-5156-4d4b-bcd6-7d6aefd29448\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4a9a21-2fszs" Sep 07 16:42:19.637463 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:19.637408 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4a9a21-2fszs" Sep 07 16:42:19.912959 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:19.912928 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4a9a21-2fszs"] Sep 07 16:42:19.915203 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:42:19.915174 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32fa202a_5156_4d4b_bcd6_7d6aefd29448.slice/crio-7cc977a493fbfcd7f68587f8c189570ec1fdd36a77e80ee952bfd1e3d985371d WatchSource:0}: Error finding container 7cc977a493fbfcd7f68587f8c189570ec1fdd36a77e80ee952bfd1e3d985371d: Status 404 returned error can't find the container with id 7cc977a493fbfcd7f68587f8c189570ec1fdd36a77e80ee952bfd1e3d985371d Sep 07 16:42:20.835217 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:20.835179 2580 generic.go:356] "Generic (PLEG): container finished" podID="32fa202a-5156-4d4b-bcd6-7d6aefd29448" containerID="7e0ae02ce73e7818cdfe0caa55ea0a9dbae1a5ba3535cf11d8359d7517b9dfd3" exitCode=0 Sep 07 16:42:20.835586 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:20.835240 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4a9a21-2fszs" event={"ID":"32fa202a-5156-4d4b-bcd6-7d6aefd29448","Type":"ContainerDied","Data":"7e0ae02ce73e7818cdfe0caa55ea0a9dbae1a5ba3535cf11d8359d7517b9dfd3"} Sep 07 16:42:20.835586 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:20.835285 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4a9a21-2fszs" event={"ID":"32fa202a-5156-4d4b-bcd6-7d6aefd29448","Type":"ContainerStarted","Data":"7cc977a493fbfcd7f68587f8c189570ec1fdd36a77e80ee952bfd1e3d985371d"} Sep 07 16:42:21.968049 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:21.968010 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4a9a21-2fszs" Sep 07 16:42:22.097422 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:22.097322 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32fa202a-5156-4d4b-bcd6-7d6aefd29448-maas-api-service-ca\") pod \"32fa202a-5156-4d4b-bcd6-7d6aefd29448\" (UID: \"32fa202a-5156-4d4b-bcd6-7d6aefd29448\") " Sep 07 16:42:22.097694 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:22.097670 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32fa202a-5156-4d4b-bcd6-7d6aefd29448-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "32fa202a-5156-4d4b-bcd6-7d6aefd29448" (UID: "32fa202a-5156-4d4b-bcd6-7d6aefd29448"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 16:42:22.198317 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:22.198257 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32fa202a-5156-4d4b-bcd6-7d6aefd29448-maas-api-service-ca\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:42:22.844596 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:22.844561 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4a9a21-2fszs" event={"ID":"32fa202a-5156-4d4b-bcd6-7d6aefd29448","Type":"ContainerDied","Data":"7cc977a493fbfcd7f68587f8c189570ec1fdd36a77e80ee952bfd1e3d985371d"} Sep 07 16:42:22.844596 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:22.844594 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4a9a21-2fszs" Sep 07 16:42:22.844784 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:22.844609 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7cc977a493fbfcd7f68587f8c189570ec1fdd36a77e80ee952bfd1e3d985371d" Sep 07 16:42:24.203106 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:24.202466 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf"] Sep 07 16:42:24.203106 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:24.202985 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed56d248-38d2-42d3-bb8c-31b9c14f3c2d" containerName="istio-proxy" Sep 07 16:42:24.203106 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:24.203008 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed56d248-38d2-42d3-bb8c-31b9c14f3c2d" containerName="istio-proxy" Sep 07 16:42:24.203106 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:24.203050 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32fa202a-5156-4d4b-bcd6-7d6aefd29448" containerName="revoke-keys" Sep 07 16:42:24.203106 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:24.203059 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="32fa202a-5156-4d4b-bcd6-7d6aefd29448" containerName="revoke-keys" Sep 07 16:42:24.203778 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:24.203157 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="32fa202a-5156-4d4b-bcd6-7d6aefd29448" containerName="revoke-keys" Sep 07 16:42:24.206075 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:24.206053 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf" Sep 07 16:42:24.208426 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:24.208404 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-4180e8-openshift-default-dockercfg-czck6\"" Sep 07 16:42:24.217845 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:24.217822 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf"] Sep 07 16:42:24.320971 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:24.320935 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4331904-45b6-4e7f-9eec-cf23d280096d-istio-envoy\") pod \"e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf\" (UID: \"f4331904-45b6-4e7f-9eec-cf23d280096d\") " pod="openshift-ingress/e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf" Sep 07 16:42:24.321176 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:24.321005 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4331904-45b6-4e7f-9eec-cf23d280096d-credential-socket\") pod \"e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf\" (UID: \"f4331904-45b6-4e7f-9eec-cf23d280096d\") " pod="openshift-ingress/e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf" Sep 07 16:42:24.321176 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:24.321060 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8gmp\" (UniqueName: \"kubernetes.io/projected/f4331904-45b6-4e7f-9eec-cf23d280096d-kube-api-access-x8gmp\") pod \"e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf\" (UID: \"f4331904-45b6-4e7f-9eec-cf23d280096d\") " pod="openshift-ingress/e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf" Sep 07 16:42:24.321176 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:24.321128 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4331904-45b6-4e7f-9eec-cf23d280096d-istio-token\") pod \"e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf\" (UID: \"f4331904-45b6-4e7f-9eec-cf23d280096d\") " pod="openshift-ingress/e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf" Sep 07 16:42:24.321335 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:24.321195 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4331904-45b6-4e7f-9eec-cf23d280096d-workload-socket\") pod \"e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf\" (UID: \"f4331904-45b6-4e7f-9eec-cf23d280096d\") " pod="openshift-ingress/e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf" Sep 07 16:42:24.321335 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:24.321256 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4331904-45b6-4e7f-9eec-cf23d280096d-workload-certs\") pod \"e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf\" (UID: \"f4331904-45b6-4e7f-9eec-cf23d280096d\") " pod="openshift-ingress/e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf" Sep 07 16:42:24.321335 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:24.321319 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4331904-45b6-4e7f-9eec-cf23d280096d-istio-podinfo\") pod \"e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf\" (UID: \"f4331904-45b6-4e7f-9eec-cf23d280096d\") " pod="openshift-ingress/e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf" Sep 07 16:42:24.321443 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:24.321353 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4331904-45b6-4e7f-9eec-cf23d280096d-istio-data\") pod \"e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf\" (UID: \"f4331904-45b6-4e7f-9eec-cf23d280096d\") " pod="openshift-ingress/e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf" Sep 07 16:42:24.321443 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:24.321393 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4331904-45b6-4e7f-9eec-cf23d280096d-istiod-ca-cert\") pod \"e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf\" (UID: \"f4331904-45b6-4e7f-9eec-cf23d280096d\") " pod="openshift-ingress/e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf" Sep 07 16:42:24.421785 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:24.421755 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4331904-45b6-4e7f-9eec-cf23d280096d-istio-envoy\") pod \"e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf\" (UID: \"f4331904-45b6-4e7f-9eec-cf23d280096d\") " pod="openshift-ingress/e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf" Sep 07 16:42:24.421950 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:24.421792 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4331904-45b6-4e7f-9eec-cf23d280096d-credential-socket\") pod \"e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf\" (UID: \"f4331904-45b6-4e7f-9eec-cf23d280096d\") " pod="openshift-ingress/e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf" Sep 07 16:42:24.421950 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:24.421824 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8gmp\" (UniqueName: \"kubernetes.io/projected/f4331904-45b6-4e7f-9eec-cf23d280096d-kube-api-access-x8gmp\") pod \"e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf\" (UID: \"f4331904-45b6-4e7f-9eec-cf23d280096d\") " pod="openshift-ingress/e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf" Sep 07 16:42:24.421950 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:24.421845 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4331904-45b6-4e7f-9eec-cf23d280096d-istio-token\") pod \"e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf\" (UID: \"f4331904-45b6-4e7f-9eec-cf23d280096d\") " pod="openshift-ingress/e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf" Sep 07 16:42:24.421950 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:24.421883 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4331904-45b6-4e7f-9eec-cf23d280096d-workload-socket\") pod \"e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf\" (UID: \"f4331904-45b6-4e7f-9eec-cf23d280096d\") " pod="openshift-ingress/e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf" Sep 07 16:42:24.421950 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:24.421929 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4331904-45b6-4e7f-9eec-cf23d280096d-workload-certs\") pod \"e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf\" (UID: \"f4331904-45b6-4e7f-9eec-cf23d280096d\") " pod="openshift-ingress/e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf" Sep 07 16:42:24.422419 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:24.421952 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4331904-45b6-4e7f-9eec-cf23d280096d-istio-podinfo\") pod \"e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf\" (UID: \"f4331904-45b6-4e7f-9eec-cf23d280096d\") " pod="openshift-ingress/e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf" Sep 07 16:42:24.422419 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:24.421973 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4331904-45b6-4e7f-9eec-cf23d280096d-istio-data\") pod \"e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf\" (UID: \"f4331904-45b6-4e7f-9eec-cf23d280096d\") " pod="openshift-ingress/e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf" Sep 07 16:42:24.422419 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:24.422002 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4331904-45b6-4e7f-9eec-cf23d280096d-istiod-ca-cert\") pod \"e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf\" (UID: \"f4331904-45b6-4e7f-9eec-cf23d280096d\") " pod="openshift-ingress/e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf" Sep 07 16:42:24.422419 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:24.422253 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4331904-45b6-4e7f-9eec-cf23d280096d-credential-socket\") pod \"e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf\" (UID: \"f4331904-45b6-4e7f-9eec-cf23d280096d\") " pod="openshift-ingress/e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf" Sep 07 16:42:24.422622 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:24.422452 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4331904-45b6-4e7f-9eec-cf23d280096d-workload-certs\") pod \"e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf\" (UID: \"f4331904-45b6-4e7f-9eec-cf23d280096d\") " pod="openshift-ingress/e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf" Sep 07 16:42:24.422622 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:24.422577 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4331904-45b6-4e7f-9eec-cf23d280096d-workload-socket\") pod \"e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf\" (UID: \"f4331904-45b6-4e7f-9eec-cf23d280096d\") " pod="openshift-ingress/e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf" Sep 07 16:42:24.422723 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:24.422705 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4331904-45b6-4e7f-9eec-cf23d280096d-istiod-ca-cert\") pod \"e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf\" (UID: \"f4331904-45b6-4e7f-9eec-cf23d280096d\") " pod="openshift-ingress/e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf" Sep 07 16:42:24.422781 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:24.422732 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4331904-45b6-4e7f-9eec-cf23d280096d-istio-data\") pod \"e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf\" (UID: \"f4331904-45b6-4e7f-9eec-cf23d280096d\") " pod="openshift-ingress/e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf" Sep 07 16:42:24.424673 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:24.424652 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4331904-45b6-4e7f-9eec-cf23d280096d-istio-envoy\") pod \"e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf\" (UID: \"f4331904-45b6-4e7f-9eec-cf23d280096d\") " pod="openshift-ingress/e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf" Sep 07 16:42:24.424798 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:24.424778 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4331904-45b6-4e7f-9eec-cf23d280096d-istio-podinfo\") pod \"e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf\" (UID: \"f4331904-45b6-4e7f-9eec-cf23d280096d\") " pod="openshift-ingress/e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf" Sep 07 16:42:24.428506 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:24.428475 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4331904-45b6-4e7f-9eec-cf23d280096d-istio-token\") pod \"e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf\" (UID: \"f4331904-45b6-4e7f-9eec-cf23d280096d\") " pod="openshift-ingress/e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf" Sep 07 16:42:24.429247 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:24.429226 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8gmp\" (UniqueName: \"kubernetes.io/projected/f4331904-45b6-4e7f-9eec-cf23d280096d-kube-api-access-x8gmp\") pod \"e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf\" (UID: \"f4331904-45b6-4e7f-9eec-cf23d280096d\") " pod="openshift-ingress/e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf" Sep 07 16:42:24.518107 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:24.518006 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf" Sep 07 16:42:24.797816 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:24.797680 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf"] Sep 07 16:42:24.800456 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:42:24.800425 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4331904_45b6_4e7f_9eec_cf23d280096d.slice/crio-9b64b712bdb6a4fdc4abe9bafb161095e24e0d43d5d0c13024e4e25107af1b5f WatchSource:0}: Error finding container 9b64b712bdb6a4fdc4abe9bafb161095e24e0d43d5d0c13024e4e25107af1b5f: Status 404 returned error can't find the container with id 9b64b712bdb6a4fdc4abe9bafb161095e24e0d43d5d0c13024e4e25107af1b5f Sep 07 16:42:24.803171 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:24.803131 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:42:24.803274 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:24.803223 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:42:24.803274 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:24.803266 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:42:24.857277 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:24.857243 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf" event={"ID":"f4331904-45b6-4e7f-9eec-cf23d280096d","Type":"ContainerStarted","Data":"9b64b712bdb6a4fdc4abe9bafb161095e24e0d43d5d0c13024e4e25107af1b5f"} Sep 07 16:42:25.863578 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:25.863536 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf" event={"ID":"f4331904-45b6-4e7f-9eec-cf23d280096d","Type":"ContainerStarted","Data":"6e59e425111da237dc999cb250f43174ec3b836723186cff581b457c65ceabe1"} Sep 07 16:42:25.884450 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:25.884392 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf" podStartSLOduration=1.884375036 podStartE2EDuration="1.884375036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:42:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:42:25.883242834 +0000 UTC m=+1374.918465930" watchObservedRunningTime="2026-09-07 16:42:25.884375036 +0000 UTC m=+1374.919598105" Sep 07 16:42:26.518220 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:26.518183 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf" Sep 07 16:42:26.523421 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:26.523397 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf" Sep 07 16:42:26.868345 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:26.868261 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf" Sep 07 16:42:26.869311 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:26.869288 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf" Sep 07 16:42:29.550384 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:29.550326 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4180e8-844b4ccfb6-xkcn5"] Sep 07 16:42:29.553900 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:29.553879 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4180e8-844b4ccfb6-xkcn5" Sep 07 16:42:29.555796 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:29.555776 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-4180e8\"" Sep 07 16:42:29.564201 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:29.564169 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4180e8-844b4ccfb6-xkcn5"] Sep 07 16:42:29.671255 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:29.671215 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl5vp\" (UniqueName: \"kubernetes.io/projected/22e1eaae-3bfa-4163-b671-6e990d721e40-kube-api-access-bl5vp\") pod \"payload-pre-processing-e2e-ipp-a-4180e8-844b4ccfb6-xkcn5\" (UID: \"22e1eaae-3bfa-4163-b671-6e990d721e40\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4180e8-844b4ccfb6-xkcn5" Sep 07 16:42:29.671434 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:29.671390 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22e1eaae-3bfa-4163-b671-6e990d721e40-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4180e8-844b4ccfb6-xkcn5\" (UID: \"22e1eaae-3bfa-4163-b671-6e990d721e40\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4180e8-844b4ccfb6-xkcn5" Sep 07 16:42:29.774012 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:29.773808 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4a9a21-698c7fd6c9-4bdr6"] Sep 07 16:42:29.774012 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:29.773959 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl5vp\" (UniqueName: \"kubernetes.io/projected/22e1eaae-3bfa-4163-b671-6e990d721e40-kube-api-access-bl5vp\") pod \"payload-pre-processing-e2e-ipp-a-4180e8-844b4ccfb6-xkcn5\" (UID: \"22e1eaae-3bfa-4163-b671-6e990d721e40\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4180e8-844b4ccfb6-xkcn5" Sep 07 16:42:29.774303 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:29.774082 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4a9a21-698c7fd6c9-4bdr6" podUID="c90db097-6a82-43c5-bcb7-2f4603f76a5c" containerName="payload-processing" containerID="cri-o://dc85920362207839b20d8d1d176c5068733bb618185bd408899d3ad04e013c46" gracePeriod=30 Sep 07 16:42:29.774632 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:29.774094 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22e1eaae-3bfa-4163-b671-6e990d721e40-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4180e8-844b4ccfb6-xkcn5\" (UID: \"22e1eaae-3bfa-4163-b671-6e990d721e40\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4180e8-844b4ccfb6-xkcn5" Sep 07 16:42:29.774988 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:29.774947 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22e1eaae-3bfa-4163-b671-6e990d721e40-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4180e8-844b4ccfb6-xkcn5\" (UID: \"22e1eaae-3bfa-4163-b671-6e990d721e40\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4180e8-844b4ccfb6-xkcn5" Sep 07 16:42:29.789391 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:29.789365 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl5vp\" (UniqueName: \"kubernetes.io/projected/22e1eaae-3bfa-4163-b671-6e990d721e40-kube-api-access-bl5vp\") pod \"payload-pre-processing-e2e-ipp-a-4180e8-844b4ccfb6-xkcn5\" (UID: \"22e1eaae-3bfa-4163-b671-6e990d721e40\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4180e8-844b4ccfb6-xkcn5" Sep 07 16:42:29.865920 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:29.865891 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4180e8-844b4ccfb6-xkcn5" Sep 07 16:42:30.040752 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:30.040726 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4a9a21-698c7fd6c9-4bdr6" Sep 07 16:42:30.077147 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:30.077057 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x7j6k\" (UniqueName: \"kubernetes.io/projected/c90db097-6a82-43c5-bcb7-2f4603f76a5c-kube-api-access-x7j6k\") pod \"c90db097-6a82-43c5-bcb7-2f4603f76a5c\" (UID: \"c90db097-6a82-43c5-bcb7-2f4603f76a5c\") " Sep 07 16:42:30.077147 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:30.077104 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c90db097-6a82-43c5-bcb7-2f4603f76a5c-plugins-config-volume\") pod \"c90db097-6a82-43c5-bcb7-2f4603f76a5c\" (UID: \"c90db097-6a82-43c5-bcb7-2f4603f76a5c\") " Sep 07 16:42:30.077574 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:30.077543 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c90db097-6a82-43c5-bcb7-2f4603f76a5c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c90db097-6a82-43c5-bcb7-2f4603f76a5c" (UID: "c90db097-6a82-43c5-bcb7-2f4603f76a5c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 16:42:30.079391 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:30.079368 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c90db097-6a82-43c5-bcb7-2f4603f76a5c-kube-api-access-x7j6k" (OuterVolumeSpecName: "kube-api-access-x7j6k") pod "c90db097-6a82-43c5-bcb7-2f4603f76a5c" (UID: "c90db097-6a82-43c5-bcb7-2f4603f76a5c"). InnerVolumeSpecName "kube-api-access-x7j6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 16:42:30.092661 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:30.092633 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4a9a21-2fszs"] Sep 07 16:42:30.097224 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:30.097202 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4a9a21-2fszs"] Sep 07 16:42:30.145703 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:30.145679 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4180e8-844b4ccfb6-xkcn5"] Sep 07 16:42:30.147930 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:42:30.147908 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22e1eaae_3bfa_4163_b671_6e990d721e40.slice/crio-a83c247eed0ffd62a1f414a10adba5f28d9886024b4748979fd6fe482ba131db WatchSource:0}: Error finding container a83c247eed0ffd62a1f414a10adba5f28d9886024b4748979fd6fe482ba131db: Status 404 returned error can't find the container with id a83c247eed0ffd62a1f414a10adba5f28d9886024b4748979fd6fe482ba131db Sep 07 16:42:30.178471 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:30.178430 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7j6k\" (UniqueName: \"kubernetes.io/projected/c90db097-6a82-43c5-bcb7-2f4603f76a5c-kube-api-access-x7j6k\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:42:30.178471 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:30.178453 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c90db097-6a82-43c5-bcb7-2f4603f76a5c-plugins-config-volume\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:42:30.889195 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:30.889157 2580 generic.go:356] "Generic (PLEG): container finished" podID="c90db097-6a82-43c5-bcb7-2f4603f76a5c" containerID="dc85920362207839b20d8d1d176c5068733bb618185bd408899d3ad04e013c46" exitCode=0 Sep 07 16:42:30.889661 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:30.889213 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4a9a21-698c7fd6c9-4bdr6" Sep 07 16:42:30.889661 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:30.889273 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4a9a21-698c7fd6c9-4bdr6" event={"ID":"c90db097-6a82-43c5-bcb7-2f4603f76a5c","Type":"ContainerDied","Data":"dc85920362207839b20d8d1d176c5068733bb618185bd408899d3ad04e013c46"} Sep 07 16:42:30.889661 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:30.889332 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4a9a21-698c7fd6c9-4bdr6" event={"ID":"c90db097-6a82-43c5-bcb7-2f4603f76a5c","Type":"ContainerDied","Data":"53e56e62908c54511c3e4a16fe6e4582a5c80fc840a50afa0f7ed9fe9e194e01"} Sep 07 16:42:30.889661 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:30.889363 2580 scope.go:117] "RemoveContainer" containerID="dc85920362207839b20d8d1d176c5068733bb618185bd408899d3ad04e013c46" Sep 07 16:42:30.891037 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:30.890990 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4180e8-844b4ccfb6-xkcn5" event={"ID":"22e1eaae-3bfa-4163-b671-6e990d721e40","Type":"ContainerStarted","Data":"b36045470286d910242b0c6c272fb479990e06a27f7d8e7cca5e6fbf71592039"} Sep 07 16:42:30.891159 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:30.891055 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4180e8-844b4ccfb6-xkcn5" event={"ID":"22e1eaae-3bfa-4163-b671-6e990d721e40","Type":"ContainerStarted","Data":"a83c247eed0ffd62a1f414a10adba5f28d9886024b4748979fd6fe482ba131db"} Sep 07 16:42:30.891379 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:30.891366 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4180e8-844b4ccfb6-xkcn5" Sep 07 16:42:30.898955 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:30.898938 2580 scope.go:117] "RemoveContainer" containerID="dc85920362207839b20d8d1d176c5068733bb618185bd408899d3ad04e013c46" Sep 07 16:42:30.899265 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:42:30.899245 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc85920362207839b20d8d1d176c5068733bb618185bd408899d3ad04e013c46\": container with ID starting with dc85920362207839b20d8d1d176c5068733bb618185bd408899d3ad04e013c46 not found: ID does not exist" containerID="dc85920362207839b20d8d1d176c5068733bb618185bd408899d3ad04e013c46" Sep 07 16:42:30.899335 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:30.899273 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc85920362207839b20d8d1d176c5068733bb618185bd408899d3ad04e013c46"} err="failed to get container status \"dc85920362207839b20d8d1d176c5068733bb618185bd408899d3ad04e013c46\": rpc error: code = NotFound desc = could not find container \"dc85920362207839b20d8d1d176c5068733bb618185bd408899d3ad04e013c46\": container with ID starting with dc85920362207839b20d8d1d176c5068733bb618185bd408899d3ad04e013c46 not found: ID does not exist" Sep 07 16:42:30.906560 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:30.906509 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4180e8-844b4ccfb6-xkcn5" podStartSLOduration=1.906494381 podStartE2EDuration="1.906494381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:42:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:42:30.904960877 +0000 UTC m=+1379.940183952" watchObservedRunningTime="2026-09-07 16:42:30.906494381 +0000 UTC m=+1379.941717452" Sep 07 16:42:30.915289 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:30.915257 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4a9a21-698c7fd6c9-4bdr6"] Sep 07 16:42:30.918128 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:30.918106 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4a9a21-698c7fd6c9-4bdr6"] Sep 07 16:42:31.419082 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:31.419053 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32fa202a-5156-4d4b-bcd6-7d6aefd29448" path="/var/lib/kubelet/pods/32fa202a-5156-4d4b-bcd6-7d6aefd29448/volumes" Sep 07 16:42:31.419407 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:31.419394 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c90db097-6a82-43c5-bcb7-2f4603f76a5c" path="/var/lib/kubelet/pods/c90db097-6a82-43c5-bcb7-2f4603f76a5c/volumes" Sep 07 16:42:41.897436 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:41.897405 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4180e8-844b4ccfb6-xkcn5" Sep 07 16:42:45.971349 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:45.971311 2580 generic.go:356] "Generic (PLEG): container finished" podID="f4331904-45b6-4e7f-9eec-cf23d280096d" containerID="6e59e425111da237dc999cb250f43174ec3b836723186cff581b457c65ceabe1" exitCode=0 Sep 07 16:42:45.971349 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:45.971358 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf" event={"ID":"f4331904-45b6-4e7f-9eec-cf23d280096d","Type":"ContainerDied","Data":"6e59e425111da237dc999cb250f43174ec3b836723186cff581b457c65ceabe1"} Sep 07 16:42:45.971839 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:45.971793 2580 scope.go:117] "RemoveContainer" containerID="6e59e425111da237dc999cb250f43174ec3b836723186cff581b457c65ceabe1" Sep 07 16:42:45.972873 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:45.972841 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:42:45.972972 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:45.972911 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:42:45.972972 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:45.972942 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 16:42:46.978377 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:46.978317 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf" event={"ID":"f4331904-45b6-4e7f-9eec-cf23d280096d","Type":"ContainerStarted","Data":"358fea81adb1bfe6cdbe09bd61e48115731a8c47a60897b5f0ac572ea41c7b75"} Sep 07 16:42:47.519134 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:47.519087 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf" Sep 07 16:42:47.519414 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:47.519383 2580 patch_prober.go:28] interesting pod/e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Sep 07 16:42:47.519526 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:47.519448 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf" podUID="f4331904-45b6-4e7f-9eec-cf23d280096d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Sep 07 16:42:48.518928 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:48.518876 2580 patch_prober.go:28] interesting pod/e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Sep 07 16:42:48.519451 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:48.518953 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf" podUID="f4331904-45b6-4e7f-9eec-cf23d280096d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Sep 07 16:42:49.519365 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:49.519313 2580 patch_prober.go:28] interesting pod/e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Sep 07 16:42:49.519896 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:49.519405 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf" podUID="f4331904-45b6-4e7f-9eec-cf23d280096d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Sep 07 16:42:50.519265 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:50.519222 2580 patch_prober.go:28] interesting pod/e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Sep 07 16:42:50.519490 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:50.519297 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf" podUID="f4331904-45b6-4e7f-9eec-cf23d280096d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Sep 07 16:42:51.519279 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:51.519242 2580 patch_prober.go:28] interesting pod/e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Sep 07 16:42:51.519616 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:51.519584 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf" podUID="f4331904-45b6-4e7f-9eec-cf23d280096d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Sep 07 16:42:52.523187 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:52.523156 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf" Sep 07 16:42:52.523599 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:52.523470 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf" Sep 07 16:42:52.524265 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:52.524251 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf" Sep 07 16:42:58.646304 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:58.646257 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6a45e8-77b64f4b9c-v6tdm"] Sep 07 16:42:58.646941 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:58.646635 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c90db097-6a82-43c5-bcb7-2f4603f76a5c" containerName="payload-processing" Sep 07 16:42:58.646941 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:58.646647 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c90db097-6a82-43c5-bcb7-2f4603f76a5c" containerName="payload-processing" Sep 07 16:42:58.646941 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:58.646702 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c90db097-6a82-43c5-bcb7-2f4603f76a5c" containerName="payload-processing" Sep 07 16:42:58.649106 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:58.649085 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a45e8-77b64f4b9c-v6tdm" Sep 07 16:42:58.651316 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:58.651294 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-6a45e8-dockercfg-pgfk9\"" Sep 07 16:42:58.651420 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:58.651353 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-6a45e8\"" Sep 07 16:42:58.661070 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:58.661041 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6a45e8-77b64f4b9c-v6tdm"] Sep 07 16:42:58.738014 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:58.737966 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz7rr\" (UniqueName: \"kubernetes.io/projected/0a837a0b-bcaf-4f08-ac76-37ec581a8ee0-kube-api-access-bz7rr\") pod \"payload-processing-e2e-ipp-b-6a45e8-77b64f4b9c-v6tdm\" (UID: \"0a837a0b-bcaf-4f08-ac76-37ec581a8ee0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6a45e8-77b64f4b9c-v6tdm" Sep 07 16:42:58.738014 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:58.738013 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a837a0b-bcaf-4f08-ac76-37ec581a8ee0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6a45e8-77b64f4b9c-v6tdm\" (UID: \"0a837a0b-bcaf-4f08-ac76-37ec581a8ee0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6a45e8-77b64f4b9c-v6tdm" Sep 07 16:42:58.839016 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:58.838975 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz7rr\" (UniqueName: \"kubernetes.io/projected/0a837a0b-bcaf-4f08-ac76-37ec581a8ee0-kube-api-access-bz7rr\") pod \"payload-processing-e2e-ipp-b-6a45e8-77b64f4b9c-v6tdm\" (UID: \"0a837a0b-bcaf-4f08-ac76-37ec581a8ee0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6a45e8-77b64f4b9c-v6tdm" Sep 07 16:42:58.839016 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:58.839037 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a837a0b-bcaf-4f08-ac76-37ec581a8ee0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6a45e8-77b64f4b9c-v6tdm\" (UID: \"0a837a0b-bcaf-4f08-ac76-37ec581a8ee0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6a45e8-77b64f4b9c-v6tdm" Sep 07 16:42:58.839772 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:58.839750 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a837a0b-bcaf-4f08-ac76-37ec581a8ee0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6a45e8-77b64f4b9c-v6tdm\" (UID: \"0a837a0b-bcaf-4f08-ac76-37ec581a8ee0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6a45e8-77b64f4b9c-v6tdm" Sep 07 16:42:58.846418 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:58.846399 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz7rr\" (UniqueName: \"kubernetes.io/projected/0a837a0b-bcaf-4f08-ac76-37ec581a8ee0-kube-api-access-bz7rr\") pod \"payload-processing-e2e-ipp-b-6a45e8-77b64f4b9c-v6tdm\" (UID: \"0a837a0b-bcaf-4f08-ac76-37ec581a8ee0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6a45e8-77b64f4b9c-v6tdm" Sep 07 16:42:58.960086 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:58.960012 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a45e8-77b64f4b9c-v6tdm" Sep 07 16:42:59.261043 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:42:59.260951 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6a45e8-77b64f4b9c-v6tdm"] Sep 07 16:42:59.263812 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:42:59.263783 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a837a0b_bcaf_4f08_ac76_37ec581a8ee0.slice/crio-d876a0fd5882a398e383d580bc365593a7f8ded0cfe56086527ef0b8ce2c49bc WatchSource:0}: Error finding container d876a0fd5882a398e383d580bc365593a7f8ded0cfe56086527ef0b8ce2c49bc: Status 404 returned error can't find the container with id d876a0fd5882a398e383d580bc365593a7f8ded0cfe56086527ef0b8ce2c49bc Sep 07 16:43:00.049240 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:43:00.049205 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a45e8-77b64f4b9c-v6tdm" event={"ID":"0a837a0b-bcaf-4f08-ac76-37ec581a8ee0","Type":"ContainerStarted","Data":"785bdfef3b128baa50849aa813b81af176fc8120e18e3a09a741a65c024b06dc"} Sep 07 16:43:00.049661 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:43:00.049260 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a45e8-77b64f4b9c-v6tdm" event={"ID":"0a837a0b-bcaf-4f08-ac76-37ec581a8ee0","Type":"ContainerStarted","Data":"d876a0fd5882a398e383d580bc365593a7f8ded0cfe56086527ef0b8ce2c49bc"} Sep 07 16:43:00.049661 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:43:00.049301 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a45e8-77b64f4b9c-v6tdm" Sep 07 16:43:00.064998 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:43:00.064944 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a45e8-77b64f4b9c-v6tdm" podStartSLOduration=2.0649290750000002 podStartE2EDuration="2.064929075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:42:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:43:00.062665283 +0000 UTC m=+1409.097888353" watchObservedRunningTime="2026-09-07 16:43:00.064929075 +0000 UTC m=+1409.100152145" Sep 07 16:43:11.056232 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:43:11.056200 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a45e8-77b64f4b9c-v6tdm" Sep 07 16:43:26.814475 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:43:26.814391 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-6a45e8-77b64f4b9c-v6tdm_0a837a0b-bcaf-4f08-ac76-37ec581a8ee0/payload-processing/0.log" Sep 07 16:44:09.877918 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:09.877815 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-6a45e8-77b64f4b9c-v6tdm_0a837a0b-bcaf-4f08-ac76-37ec581a8ee0/payload-processing/0.log" Sep 07 16:44:15.025943 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:15.025906 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1cc79d-7d58cb6f74-8vwqn"] Sep 07 16:44:15.029284 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:15.029264 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1cc79d-7d58cb6f74-8vwqn" Sep 07 16:44:15.031553 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:15.031529 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-1cc79d\"" Sep 07 16:44:15.038238 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:15.038212 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1cc79d-7d58cb6f74-8vwqn"] Sep 07 16:44:15.059608 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:15.059572 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-1cc79d-74598dfd45-k6pnb"] Sep 07 16:44:15.063109 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:15.063083 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1cc79d-74598dfd45-k6pnb" Sep 07 16:44:15.065097 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:15.065065 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-1cc79d-dockercfg-dnqzm\"" Sep 07 16:44:15.074180 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:15.074156 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-1cc79d-74598dfd45-k6pnb"] Sep 07 16:44:15.145735 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:15.145695 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a25bd5a5-a85b-47a7-9b07-5a63adf117f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-1cc79d-7d58cb6f74-8vwqn\" (UID: \"a25bd5a5-a85b-47a7-9b07-5a63adf117f5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1cc79d-7d58cb6f74-8vwqn" Sep 07 16:44:15.145735 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:15.145735 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqgnf\" (UniqueName: \"kubernetes.io/projected/a25bd5a5-a85b-47a7-9b07-5a63adf117f5-kube-api-access-wqgnf\") pod \"payload-pre-processing-e2e-ipp-cleanup-1cc79d-7d58cb6f74-8vwqn\" (UID: \"a25bd5a5-a85b-47a7-9b07-5a63adf117f5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1cc79d-7d58cb6f74-8vwqn" Sep 07 16:44:15.146015 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:15.145857 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj7rb\" (UniqueName: \"kubernetes.io/projected/b3c21fb3-7abe-4857-b6f7-d9aa56c78cbd-kube-api-access-bj7rb\") pod \"payload-processing-e2e-ipp-cleanup-1cc79d-74598dfd45-k6pnb\" (UID: \"b3c21fb3-7abe-4857-b6f7-d9aa56c78cbd\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1cc79d-74598dfd45-k6pnb" Sep 07 16:44:15.146015 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:15.145919 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3c21fb3-7abe-4857-b6f7-d9aa56c78cbd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-1cc79d-74598dfd45-k6pnb\" (UID: \"b3c21fb3-7abe-4857-b6f7-d9aa56c78cbd\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1cc79d-74598dfd45-k6pnb" Sep 07 16:44:15.247161 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:15.247124 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj7rb\" (UniqueName: \"kubernetes.io/projected/b3c21fb3-7abe-4857-b6f7-d9aa56c78cbd-kube-api-access-bj7rb\") pod \"payload-processing-e2e-ipp-cleanup-1cc79d-74598dfd45-k6pnb\" (UID: \"b3c21fb3-7abe-4857-b6f7-d9aa56c78cbd\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1cc79d-74598dfd45-k6pnb" Sep 07 16:44:15.247376 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:15.247188 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3c21fb3-7abe-4857-b6f7-d9aa56c78cbd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-1cc79d-74598dfd45-k6pnb\" (UID: \"b3c21fb3-7abe-4857-b6f7-d9aa56c78cbd\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1cc79d-74598dfd45-k6pnb" Sep 07 16:44:15.247376 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:15.247294 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a25bd5a5-a85b-47a7-9b07-5a63adf117f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-1cc79d-7d58cb6f74-8vwqn\" (UID: \"a25bd5a5-a85b-47a7-9b07-5a63adf117f5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1cc79d-7d58cb6f74-8vwqn" Sep 07 16:44:15.247376 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:15.247320 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqgnf\" (UniqueName: \"kubernetes.io/projected/a25bd5a5-a85b-47a7-9b07-5a63adf117f5-kube-api-access-wqgnf\") pod \"payload-pre-processing-e2e-ipp-cleanup-1cc79d-7d58cb6f74-8vwqn\" (UID: \"a25bd5a5-a85b-47a7-9b07-5a63adf117f5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1cc79d-7d58cb6f74-8vwqn" Sep 07 16:44:15.247909 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:15.247884 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3c21fb3-7abe-4857-b6f7-d9aa56c78cbd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-1cc79d-74598dfd45-k6pnb\" (UID: \"b3c21fb3-7abe-4857-b6f7-d9aa56c78cbd\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1cc79d-74598dfd45-k6pnb" Sep 07 16:44:15.247974 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:15.247942 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a25bd5a5-a85b-47a7-9b07-5a63adf117f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-1cc79d-7d58cb6f74-8vwqn\" (UID: \"a25bd5a5-a85b-47a7-9b07-5a63adf117f5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1cc79d-7d58cb6f74-8vwqn" Sep 07 16:44:15.254722 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:15.254695 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj7rb\" (UniqueName: \"kubernetes.io/projected/b3c21fb3-7abe-4857-b6f7-d9aa56c78cbd-kube-api-access-bj7rb\") pod \"payload-processing-e2e-ipp-cleanup-1cc79d-74598dfd45-k6pnb\" (UID: \"b3c21fb3-7abe-4857-b6f7-d9aa56c78cbd\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1cc79d-74598dfd45-k6pnb" Sep 07 16:44:15.254830 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:15.254801 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqgnf\" (UniqueName: \"kubernetes.io/projected/a25bd5a5-a85b-47a7-9b07-5a63adf117f5-kube-api-access-wqgnf\") pod \"payload-pre-processing-e2e-ipp-cleanup-1cc79d-7d58cb6f74-8vwqn\" (UID: \"a25bd5a5-a85b-47a7-9b07-5a63adf117f5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1cc79d-7d58cb6f74-8vwqn" Sep 07 16:44:15.341010 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:15.340926 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1cc79d-7d58cb6f74-8vwqn" Sep 07 16:44:15.374902 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:15.374870 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1cc79d-74598dfd45-k6pnb" Sep 07 16:44:15.612014 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:15.611989 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1cc79d-7d58cb6f74-8vwqn"] Sep 07 16:44:15.614367 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:44:15.614318 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda25bd5a5_a85b_47a7_9b07_5a63adf117f5.slice/crio-202178b8e4e998ce12acc64295cc306cd54745c33627388053c76763fc0eaba3 WatchSource:0}: Error finding container 202178b8e4e998ce12acc64295cc306cd54745c33627388053c76763fc0eaba3: Status 404 returned error can't find the container with id 202178b8e4e998ce12acc64295cc306cd54745c33627388053c76763fc0eaba3 Sep 07 16:44:15.654008 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:15.653985 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-1cc79d-74598dfd45-k6pnb"] Sep 07 16:44:15.661085 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:44:15.661057 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3c21fb3_7abe_4857_b6f7_d9aa56c78cbd.slice/crio-0fb1651cfe33b6f6551e4e18422d643a9b07a10c04d1e78d9aaf5de3ad70dff7 WatchSource:0}: Error finding container 0fb1651cfe33b6f6551e4e18422d643a9b07a10c04d1e78d9aaf5de3ad70dff7: Status 404 returned error can't find the container with id 0fb1651cfe33b6f6551e4e18422d643a9b07a10c04d1e78d9aaf5de3ad70dff7 Sep 07 16:44:16.419850 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:16.419802 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1cc79d-7d58cb6f74-8vwqn" event={"ID":"a25bd5a5-a85b-47a7-9b07-5a63adf117f5","Type":"ContainerStarted","Data":"217644e1e612b988c2be8d74b1834583bc55085adad6434beaf38154edba596a"} Sep 07 16:44:16.420354 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:16.419861 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1cc79d-7d58cb6f74-8vwqn" event={"ID":"a25bd5a5-a85b-47a7-9b07-5a63adf117f5","Type":"ContainerStarted","Data":"202178b8e4e998ce12acc64295cc306cd54745c33627388053c76763fc0eaba3"} Sep 07 16:44:16.420354 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:16.419922 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1cc79d-7d58cb6f74-8vwqn" Sep 07 16:44:16.421484 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:16.421459 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1cc79d-74598dfd45-k6pnb" event={"ID":"b3c21fb3-7abe-4857-b6f7-d9aa56c78cbd","Type":"ContainerStarted","Data":"6a5a5db3d4c81a0c32e8ea170d568b1375627734c56b8f0634c2662283afcfaf"} Sep 07 16:44:16.421608 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:16.421493 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1cc79d-74598dfd45-k6pnb" event={"ID":"b3c21fb3-7abe-4857-b6f7-d9aa56c78cbd","Type":"ContainerStarted","Data":"0fb1651cfe33b6f6551e4e18422d643a9b07a10c04d1e78d9aaf5de3ad70dff7"} Sep 07 16:44:16.421608 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:16.421514 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1cc79d-74598dfd45-k6pnb" Sep 07 16:44:16.436807 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:16.436733 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1cc79d-7d58cb6f74-8vwqn" podStartSLOduration=1.436712591 podStartE2EDuration="1.436712591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:44:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:44:16.434474017 +0000 UTC m=+1485.469697090" watchObservedRunningTime="2026-09-07 16:44:16.436712591 +0000 UTC m=+1485.471935661" Sep 07 16:44:16.449922 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:16.449862 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1cc79d-74598dfd45-k6pnb" podStartSLOduration=1.4498479309999999 podStartE2EDuration="1.449847931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:44:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:44:16.448071254 +0000 UTC m=+1485.483294325" watchObservedRunningTime="2026-09-07 16:44:16.449847931 +0000 UTC m=+1485.485070999" Sep 07 16:44:27.431821 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:27.431790 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1cc79d-74598dfd45-k6pnb" Sep 07 16:44:27.432258 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:27.431872 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1cc79d-7d58cb6f74-8vwqn" Sep 07 16:44:31.698210 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:31.698179 2580 scope.go:117] "RemoveContainer" containerID="7d0e15fc995b4c368b5ef7b0a8d9bcb8c5a6afe5593a255af29d76d73fbf841c" Sep 07 16:44:44.917181 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:44.917142 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-1cc79d-74598dfd45-k6pnb"] Sep 07 16:44:44.917683 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:44.917402 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1cc79d-74598dfd45-k6pnb" podUID="b3c21fb3-7abe-4857-b6f7-d9aa56c78cbd" containerName="payload-processing" containerID="cri-o://6a5a5db3d4c81a0c32e8ea170d568b1375627734c56b8f0634c2662283afcfaf" gracePeriod=30 Sep 07 16:44:44.926943 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:44.926911 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1cc79d-7d58cb6f74-8vwqn"] Sep 07 16:44:44.927206 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:44.927182 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1cc79d-7d58cb6f74-8vwqn" podUID="a25bd5a5-a85b-47a7-9b07-5a63adf117f5" containerName="payload-pre-processing" containerID="cri-o://217644e1e612b988c2be8d74b1834583bc55085adad6434beaf38154edba596a" gracePeriod=30 Sep 07 16:44:45.209232 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:45.209206 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1cc79d-74598dfd45-k6pnb" Sep 07 16:44:45.212312 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:45.212292 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1cc79d-7d58cb6f74-8vwqn" Sep 07 16:44:45.235707 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:45.235677 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a25bd5a5-a85b-47a7-9b07-5a63adf117f5-plugins-config-volume\") pod \"a25bd5a5-a85b-47a7-9b07-5a63adf117f5\" (UID: \"a25bd5a5-a85b-47a7-9b07-5a63adf117f5\") " Sep 07 16:44:45.235853 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:45.235742 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3c21fb3-7abe-4857-b6f7-d9aa56c78cbd-plugins-config-volume\") pod \"b3c21fb3-7abe-4857-b6f7-d9aa56c78cbd\" (UID: \"b3c21fb3-7abe-4857-b6f7-d9aa56c78cbd\") " Sep 07 16:44:45.235853 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:45.235777 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bj7rb\" (UniqueName: \"kubernetes.io/projected/b3c21fb3-7abe-4857-b6f7-d9aa56c78cbd-kube-api-access-bj7rb\") pod \"b3c21fb3-7abe-4857-b6f7-d9aa56c78cbd\" (UID: \"b3c21fb3-7abe-4857-b6f7-d9aa56c78cbd\") " Sep 07 16:44:45.235989 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:45.235871 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wqgnf\" (UniqueName: \"kubernetes.io/projected/a25bd5a5-a85b-47a7-9b07-5a63adf117f5-kube-api-access-wqgnf\") pod \"a25bd5a5-a85b-47a7-9b07-5a63adf117f5\" (UID: \"a25bd5a5-a85b-47a7-9b07-5a63adf117f5\") " Sep 07 16:44:45.236121 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:45.236092 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a25bd5a5-a85b-47a7-9b07-5a63adf117f5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a25bd5a5-a85b-47a7-9b07-5a63adf117f5" (UID: "a25bd5a5-a85b-47a7-9b07-5a63adf117f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 16:44:45.236190 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:45.236168 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3c21fb3-7abe-4857-b6f7-d9aa56c78cbd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b3c21fb3-7abe-4857-b6f7-d9aa56c78cbd" (UID: "b3c21fb3-7abe-4857-b6f7-d9aa56c78cbd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 16:44:45.236237 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:45.236198 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a25bd5a5-a85b-47a7-9b07-5a63adf117f5-plugins-config-volume\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:44:45.238134 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:45.238109 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a25bd5a5-a85b-47a7-9b07-5a63adf117f5-kube-api-access-wqgnf" (OuterVolumeSpecName: "kube-api-access-wqgnf") pod "a25bd5a5-a85b-47a7-9b07-5a63adf117f5" (UID: "a25bd5a5-a85b-47a7-9b07-5a63adf117f5"). InnerVolumeSpecName "kube-api-access-wqgnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 16:44:45.238227 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:45.238180 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3c21fb3-7abe-4857-b6f7-d9aa56c78cbd-kube-api-access-bj7rb" (OuterVolumeSpecName: "kube-api-access-bj7rb") pod "b3c21fb3-7abe-4857-b6f7-d9aa56c78cbd" (UID: "b3c21fb3-7abe-4857-b6f7-d9aa56c78cbd"). InnerVolumeSpecName "kube-api-access-bj7rb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 16:44:45.336917 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:45.336885 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3c21fb3-7abe-4857-b6f7-d9aa56c78cbd-plugins-config-volume\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:44:45.336917 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:45.336911 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bj7rb\" (UniqueName: \"kubernetes.io/projected/b3c21fb3-7abe-4857-b6f7-d9aa56c78cbd-kube-api-access-bj7rb\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:44:45.336917 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:45.336920 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqgnf\" (UniqueName: \"kubernetes.io/projected/a25bd5a5-a85b-47a7-9b07-5a63adf117f5-kube-api-access-wqgnf\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:44:45.568774 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:45.568742 2580 generic.go:356] "Generic (PLEG): container finished" podID="a25bd5a5-a85b-47a7-9b07-5a63adf117f5" containerID="217644e1e612b988c2be8d74b1834583bc55085adad6434beaf38154edba596a" exitCode=0 Sep 07 16:44:45.568942 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:45.568785 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1cc79d-7d58cb6f74-8vwqn" Sep 07 16:44:45.568942 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:45.568822 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1cc79d-7d58cb6f74-8vwqn" event={"ID":"a25bd5a5-a85b-47a7-9b07-5a63adf117f5","Type":"ContainerDied","Data":"217644e1e612b988c2be8d74b1834583bc55085adad6434beaf38154edba596a"} Sep 07 16:44:45.568942 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:45.568881 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1cc79d-7d58cb6f74-8vwqn" event={"ID":"a25bd5a5-a85b-47a7-9b07-5a63adf117f5","Type":"ContainerDied","Data":"202178b8e4e998ce12acc64295cc306cd54745c33627388053c76763fc0eaba3"} Sep 07 16:44:45.568942 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:45.568905 2580 scope.go:117] "RemoveContainer" containerID="217644e1e612b988c2be8d74b1834583bc55085adad6434beaf38154edba596a" Sep 07 16:44:45.570183 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:45.570156 2580 generic.go:356] "Generic (PLEG): container finished" podID="b3c21fb3-7abe-4857-b6f7-d9aa56c78cbd" containerID="6a5a5db3d4c81a0c32e8ea170d568b1375627734c56b8f0634c2662283afcfaf" exitCode=0 Sep 07 16:44:45.570302 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:45.570187 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1cc79d-74598dfd45-k6pnb" event={"ID":"b3c21fb3-7abe-4857-b6f7-d9aa56c78cbd","Type":"ContainerDied","Data":"6a5a5db3d4c81a0c32e8ea170d568b1375627734c56b8f0634c2662283afcfaf"} Sep 07 16:44:45.570302 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:45.570203 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1cc79d-74598dfd45-k6pnb" Sep 07 16:44:45.570302 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:45.570235 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1cc79d-74598dfd45-k6pnb" event={"ID":"b3c21fb3-7abe-4857-b6f7-d9aa56c78cbd","Type":"ContainerDied","Data":"0fb1651cfe33b6f6551e4e18422d643a9b07a10c04d1e78d9aaf5de3ad70dff7"} Sep 07 16:44:45.579706 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:45.579691 2580 scope.go:117] "RemoveContainer" containerID="217644e1e612b988c2be8d74b1834583bc55085adad6434beaf38154edba596a" Sep 07 16:44:45.580005 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:44:45.579973 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"217644e1e612b988c2be8d74b1834583bc55085adad6434beaf38154edba596a\": container with ID starting with 217644e1e612b988c2be8d74b1834583bc55085adad6434beaf38154edba596a not found: ID does not exist" containerID="217644e1e612b988c2be8d74b1834583bc55085adad6434beaf38154edba596a" Sep 07 16:44:45.580120 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:45.580005 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"217644e1e612b988c2be8d74b1834583bc55085adad6434beaf38154edba596a"} err="failed to get container status \"217644e1e612b988c2be8d74b1834583bc55085adad6434beaf38154edba596a\": rpc error: code = NotFound desc = could not find container \"217644e1e612b988c2be8d74b1834583bc55085adad6434beaf38154edba596a\": container with ID starting with 217644e1e612b988c2be8d74b1834583bc55085adad6434beaf38154edba596a not found: ID does not exist" Sep 07 16:44:45.580120 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:45.580043 2580 scope.go:117] "RemoveContainer" containerID="6a5a5db3d4c81a0c32e8ea170d568b1375627734c56b8f0634c2662283afcfaf" Sep 07 16:44:45.587353 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:45.587329 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1cc79d-7d58cb6f74-8vwqn"] Sep 07 16:44:45.588606 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:45.588591 2580 scope.go:117] "RemoveContainer" containerID="6a5a5db3d4c81a0c32e8ea170d568b1375627734c56b8f0634c2662283afcfaf" Sep 07 16:44:45.588874 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:44:45.588852 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a5a5db3d4c81a0c32e8ea170d568b1375627734c56b8f0634c2662283afcfaf\": container with ID starting with 6a5a5db3d4c81a0c32e8ea170d568b1375627734c56b8f0634c2662283afcfaf not found: ID does not exist" containerID="6a5a5db3d4c81a0c32e8ea170d568b1375627734c56b8f0634c2662283afcfaf" Sep 07 16:44:45.588937 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:45.588890 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a5a5db3d4c81a0c32e8ea170d568b1375627734c56b8f0634c2662283afcfaf"} err="failed to get container status \"6a5a5db3d4c81a0c32e8ea170d568b1375627734c56b8f0634c2662283afcfaf\": rpc error: code = NotFound desc = could not find container \"6a5a5db3d4c81a0c32e8ea170d568b1375627734c56b8f0634c2662283afcfaf\": container with ID starting with 6a5a5db3d4c81a0c32e8ea170d568b1375627734c56b8f0634c2662283afcfaf not found: ID does not exist" Sep 07 16:44:45.589289 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:45.589272 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1cc79d-7d58cb6f74-8vwqn"] Sep 07 16:44:45.596780 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:45.596761 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-1cc79d-74598dfd45-k6pnb"] Sep 07 16:44:45.604517 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:45.604497 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-1cc79d-74598dfd45-k6pnb"] Sep 07 16:44:47.417993 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:47.417959 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a25bd5a5-a85b-47a7-9b07-5a63adf117f5" path="/var/lib/kubelet/pods/a25bd5a5-a85b-47a7-9b07-5a63adf117f5/volumes" Sep 07 16:44:47.418475 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:47.418397 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3c21fb3-7abe-4857-b6f7-d9aa56c78cbd" path="/var/lib/kubelet/pods/b3c21fb3-7abe-4857-b6f7-d9aa56c78cbd/volumes" Sep 07 16:44:54.681592 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:54.681511 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4180e8-9wm8z"] Sep 07 16:44:54.681965 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:54.681847 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a25bd5a5-a85b-47a7-9b07-5a63adf117f5" containerName="payload-pre-processing" Sep 07 16:44:54.681965 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:54.681858 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a25bd5a5-a85b-47a7-9b07-5a63adf117f5" containerName="payload-pre-processing" Sep 07 16:44:54.681965 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:54.681872 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3c21fb3-7abe-4857-b6f7-d9aa56c78cbd" containerName="payload-processing" Sep 07 16:44:54.681965 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:54.681879 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3c21fb3-7abe-4857-b6f7-d9aa56c78cbd" containerName="payload-processing" Sep 07 16:44:54.681965 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:54.681935 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a25bd5a5-a85b-47a7-9b07-5a63adf117f5" containerName="payload-pre-processing" Sep 07 16:44:54.681965 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:54.681944 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3c21fb3-7abe-4857-b6f7-d9aa56c78cbd" containerName="payload-processing" Sep 07 16:44:54.686412 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:54.686395 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4180e8-9wm8z" Sep 07 16:44:54.688641 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:54.688604 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 07 16:44:54.688838 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:54.688606 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jpjxt\"" Sep 07 16:44:54.695609 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:54.695573 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4180e8-9wm8z"] Sep 07 16:44:54.718240 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:54.718202 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0036d55a-4185-420c-9a0f-7c195bf806fc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4180e8-9wm8z\" (UID: \"0036d55a-4185-420c-9a0f-7c195bf806fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4180e8-9wm8z" Sep 07 16:44:54.818963 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:54.818921 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0036d55a-4185-420c-9a0f-7c195bf806fc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4180e8-9wm8z\" (UID: \"0036d55a-4185-420c-9a0f-7c195bf806fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4180e8-9wm8z" Sep 07 16:44:54.819561 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:54.819539 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0036d55a-4185-420c-9a0f-7c195bf806fc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4180e8-9wm8z\" (UID: \"0036d55a-4185-420c-9a0f-7c195bf806fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4180e8-9wm8z" Sep 07 16:44:55.004233 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:55.004197 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4180e8-9wm8z" Sep 07 16:44:55.272686 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:55.272645 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4180e8-9wm8z"] Sep 07 16:44:55.274898 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:44:55.274873 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0036d55a_4185_420c_9a0f_7c195bf806fc.slice/crio-66f6409ce69dbbb50cdbb7a1b0651289340cb5e57e646c1236f66a691464d7ae WatchSource:0}: Error finding container 66f6409ce69dbbb50cdbb7a1b0651289340cb5e57e646c1236f66a691464d7ae: Status 404 returned error can't find the container with id 66f6409ce69dbbb50cdbb7a1b0651289340cb5e57e646c1236f66a691464d7ae Sep 07 16:44:55.620839 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:55.620739 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4180e8-9wm8z" event={"ID":"0036d55a-4185-420c-9a0f-7c195bf806fc","Type":"ContainerStarted","Data":"40ba1106974e392e7ea12034d2a3e01e0f98028e2b4d2a1a36ae336b8690d957"} Sep 07 16:44:55.620839 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:55.620813 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4180e8-9wm8z" event={"ID":"0036d55a-4185-420c-9a0f-7c195bf806fc","Type":"ContainerStarted","Data":"66f6409ce69dbbb50cdbb7a1b0651289340cb5e57e646c1236f66a691464d7ae"} Sep 07 16:44:55.649985 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:55.649931 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4180e8-9wm8z" podStartSLOduration=1.649913051 podStartE2EDuration="1.649913051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:44:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:44:55.647816726 +0000 UTC m=+1524.683039789" watchObservedRunningTime="2026-09-07 16:44:55.649913051 +0000 UTC m=+1524.685136120" Sep 07 16:44:56.626643 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:56.626607 2580 generic.go:356] "Generic (PLEG): container finished" podID="0036d55a-4185-420c-9a0f-7c195bf806fc" containerID="40ba1106974e392e7ea12034d2a3e01e0f98028e2b4d2a1a36ae336b8690d957" exitCode=0 Sep 07 16:44:56.627078 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:56.626688 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4180e8-9wm8z" event={"ID":"0036d55a-4185-420c-9a0f-7c195bf806fc","Type":"ContainerDied","Data":"40ba1106974e392e7ea12034d2a3e01e0f98028e2b4d2a1a36ae336b8690d957"} Sep 07 16:44:57.762854 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:57.762830 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4180e8-9wm8z" Sep 07 16:44:57.846147 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:57.846111 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0036d55a-4185-420c-9a0f-7c195bf806fc-maas-api-service-ca\") pod \"0036d55a-4185-420c-9a0f-7c195bf806fc\" (UID: \"0036d55a-4185-420c-9a0f-7c195bf806fc\") " Sep 07 16:44:57.846510 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:57.846488 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0036d55a-4185-420c-9a0f-7c195bf806fc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0036d55a-4185-420c-9a0f-7c195bf806fc" (UID: "0036d55a-4185-420c-9a0f-7c195bf806fc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 16:44:57.947118 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:57.947076 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0036d55a-4185-420c-9a0f-7c195bf806fc-maas-api-service-ca\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:44:58.638016 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:58.637990 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4180e8-9wm8z" Sep 07 16:44:58.638207 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:58.638049 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4180e8-9wm8z" event={"ID":"0036d55a-4185-420c-9a0f-7c195bf806fc","Type":"ContainerDied","Data":"66f6409ce69dbbb50cdbb7a1b0651289340cb5e57e646c1236f66a691464d7ae"} Sep 07 16:44:58.638207 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:44:58.638094 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66f6409ce69dbbb50cdbb7a1b0651289340cb5e57e646c1236f66a691464d7ae" Sep 07 16:45:00.135333 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:00.135299 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8d59e2-29813325-99pb7"] Sep 07 16:45:00.135741 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:00.135721 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0036d55a-4185-420c-9a0f-7c195bf806fc" containerName="revoke-keys" Sep 07 16:45:00.135741 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:00.135740 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0036d55a-4185-420c-9a0f-7c195bf806fc" containerName="revoke-keys" Sep 07 16:45:00.135824 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:00.135814 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0036d55a-4185-420c-9a0f-7c195bf806fc" containerName="revoke-keys" Sep 07 16:45:00.138748 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:00.138732 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6a45e8-29813325-r5qbj"] Sep 07 16:45:00.138870 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:00.138853 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8d59e2-29813325-99pb7" Sep 07 16:45:00.141112 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:00.141092 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jpjxt\"" Sep 07 16:45:00.141699 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:00.141685 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7aca0f-29813325-knggl"] Sep 07 16:45:00.141816 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:00.141803 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6a45e8-29813325-r5qbj" Sep 07 16:45:00.147145 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:00.147121 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8d59e2-29813325-99pb7"] Sep 07 16:45:00.147392 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:00.147378 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7aca0f-29813325-knggl" Sep 07 16:45:00.147601 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:00.147589 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6a45e8-29813325-r5qbj"] Sep 07 16:45:00.150194 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:00.150174 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7aca0f-29813325-knggl"] Sep 07 16:45:00.154521 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:00.154499 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8d59e2-29813325-99pb7" Sep 07 16:45:00.164851 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:00.164828 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6a45e8-29813325-r5qbj" Sep 07 16:45:00.203276 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:00.203251 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7aca0f-29813325-knggl" Sep 07 16:45:00.438185 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:00.438124 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8d59e2-29813325-99pb7"] Sep 07 16:45:00.440777 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:45:00.440743 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e8d6618_66bd_40c5_981a_b3f5667f9ef2.slice/crio-d0ffe2f64128b62dd7350259cbc1a7aa2ce3cd01bee3dc474ac57274bc9a7fe4 WatchSource:0}: Error finding container d0ffe2f64128b62dd7350259cbc1a7aa2ce3cd01bee3dc474ac57274bc9a7fe4: Status 404 returned error can't find the container with id d0ffe2f64128b62dd7350259cbc1a7aa2ce3cd01bee3dc474ac57274bc9a7fe4 Sep 07 16:45:00.457717 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:00.457666 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6a45e8-29813325-r5qbj"] Sep 07 16:45:00.460184 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:45:00.460148 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02acb81e_f655_48b7_9163_010a2891ce7d.slice/crio-9024c1eb3a5cef5f451638d4b10bb01af7adc66c7f79a0770e0ef998f729d52d WatchSource:0}: Error finding container 9024c1eb3a5cef5f451638d4b10bb01af7adc66c7f79a0770e0ef998f729d52d: Status 404 returned error can't find the container with id 9024c1eb3a5cef5f451638d4b10bb01af7adc66c7f79a0770e0ef998f729d52d Sep 07 16:45:00.500186 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:00.500158 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7aca0f-29813325-knggl"] Sep 07 16:45:00.502896 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:45:00.502875 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbcf587a_f5c9_4479_917d_1055a0eb395a.slice/crio-aed0cc2adbababd4d00c412c0c0a3177913213778db2a5da189dc2dbc7b91eea WatchSource:0}: Error finding container aed0cc2adbababd4d00c412c0c0a3177913213778db2a5da189dc2dbc7b91eea: Status 404 returned error can't find the container with id aed0cc2adbababd4d00c412c0c0a3177913213778db2a5da189dc2dbc7b91eea Sep 07 16:45:00.649258 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:00.649226 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8d59e2-29813325-99pb7" event={"ID":"1e8d6618-66bd-40c5-981a-b3f5667f9ef2","Type":"ContainerStarted","Data":"d0ffe2f64128b62dd7350259cbc1a7aa2ce3cd01bee3dc474ac57274bc9a7fe4"} Sep 07 16:45:00.650412 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:00.650381 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7aca0f-29813325-knggl" event={"ID":"bbcf587a-f5c9-4479-917d-1055a0eb395a","Type":"ContainerStarted","Data":"aed0cc2adbababd4d00c412c0c0a3177913213778db2a5da189dc2dbc7b91eea"} Sep 07 16:45:00.651579 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:00.651557 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6a45e8-29813325-r5qbj" event={"ID":"02acb81e-f655-48b7-9163-010a2891ce7d","Type":"ContainerStarted","Data":"9024c1eb3a5cef5f451638d4b10bb01af7adc66c7f79a0770e0ef998f729d52d"} Sep 07 16:45:01.658902 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:01.658808 2580 generic.go:356] "Generic (PLEG): container finished" podID="1e8d6618-66bd-40c5-981a-b3f5667f9ef2" containerID="f587351a838ef8c64a532a8f0886cf023eaf1579ed5a6ce3e151a1fb3dbc69c7" exitCode=0 Sep 07 16:45:01.659359 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:01.658887 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8d59e2-29813325-99pb7" event={"ID":"1e8d6618-66bd-40c5-981a-b3f5667f9ef2","Type":"ContainerDied","Data":"f587351a838ef8c64a532a8f0886cf023eaf1579ed5a6ce3e151a1fb3dbc69c7"} Sep 07 16:45:01.660446 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:01.660422 2580 generic.go:356] "Generic (PLEG): container finished" podID="bbcf587a-f5c9-4479-917d-1055a0eb395a" containerID="99902eb549123a8a11f3d0911452f8ede9314c51897dd4e2920c08d72cc5aff7" exitCode=0 Sep 07 16:45:01.660559 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:01.660503 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7aca0f-29813325-knggl" event={"ID":"bbcf587a-f5c9-4479-917d-1055a0eb395a","Type":"ContainerDied","Data":"99902eb549123a8a11f3d0911452f8ede9314c51897dd4e2920c08d72cc5aff7"} Sep 07 16:45:01.662000 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:01.661980 2580 generic.go:356] "Generic (PLEG): container finished" podID="02acb81e-f655-48b7-9163-010a2891ce7d" containerID="92c4c83723e8466ff7a66250fd524cea6c98dbc702872c03a362c326f712cccb" exitCode=0 Sep 07 16:45:01.662104 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:01.662051 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6a45e8-29813325-r5qbj" event={"ID":"02acb81e-f655-48b7-9163-010a2891ce7d","Type":"ContainerDied","Data":"92c4c83723e8466ff7a66250fd524cea6c98dbc702872c03a362c326f712cccb"} Sep 07 16:45:02.811690 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:02.811664 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7aca0f-29813325-knggl" Sep 07 16:45:02.855102 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:02.855080 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8d59e2-29813325-99pb7" Sep 07 16:45:02.858085 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:02.858065 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6a45e8-29813325-r5qbj" Sep 07 16:45:03.675034 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:03.674983 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8d59e2-29813325-99pb7" Sep 07 16:45:03.675034 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:03.674998 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8d59e2-29813325-99pb7" event={"ID":"1e8d6618-66bd-40c5-981a-b3f5667f9ef2","Type":"ContainerDied","Data":"d0ffe2f64128b62dd7350259cbc1a7aa2ce3cd01bee3dc474ac57274bc9a7fe4"} Sep 07 16:45:03.675287 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:03.675069 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0ffe2f64128b62dd7350259cbc1a7aa2ce3cd01bee3dc474ac57274bc9a7fe4" Sep 07 16:45:03.676583 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:03.676521 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7aca0f-29813325-knggl" Sep 07 16:45:03.676583 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:03.676540 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7aca0f-29813325-knggl" event={"ID":"bbcf587a-f5c9-4479-917d-1055a0eb395a","Type":"ContainerDied","Data":"aed0cc2adbababd4d00c412c0c0a3177913213778db2a5da189dc2dbc7b91eea"} Sep 07 16:45:03.676583 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:03.676576 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aed0cc2adbababd4d00c412c0c0a3177913213778db2a5da189dc2dbc7b91eea" Sep 07 16:45:03.678754 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:03.678723 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6a45e8-29813325-r5qbj" Sep 07 16:45:03.678895 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:03.678762 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6a45e8-29813325-r5qbj" event={"ID":"02acb81e-f655-48b7-9163-010a2891ce7d","Type":"ContainerDied","Data":"9024c1eb3a5cef5f451638d4b10bb01af7adc66c7f79a0770e0ef998f729d52d"} Sep 07 16:45:03.678895 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:03.678797 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9024c1eb3a5cef5f451638d4b10bb01af7adc66c7f79a0770e0ef998f729d52d" Sep 07 16:45:10.298834 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:10.298791 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4180e8-844b4ccfb6-xkcn5"] Sep 07 16:45:10.299325 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:10.299071 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4180e8-844b4ccfb6-xkcn5" podUID="22e1eaae-3bfa-4163-b671-6e990d721e40" containerName="payload-pre-processing" containerID="cri-o://b36045470286d910242b0c6c272fb479990e06a27f7d8e7cca5e6fbf71592039" gracePeriod=30 Sep 07 16:45:10.441401 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:10.441366 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4180e8-9wm8z"] Sep 07 16:45:10.443925 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:10.443895 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4180e8-9wm8z"] Sep 07 16:45:10.560275 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:10.560205 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4180e8-844b4ccfb6-xkcn5" Sep 07 16:45:10.665908 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:10.665867 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bl5vp\" (UniqueName: \"kubernetes.io/projected/22e1eaae-3bfa-4163-b671-6e990d721e40-kube-api-access-bl5vp\") pod \"22e1eaae-3bfa-4163-b671-6e990d721e40\" (UID: \"22e1eaae-3bfa-4163-b671-6e990d721e40\") " Sep 07 16:45:10.666114 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:10.665922 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22e1eaae-3bfa-4163-b671-6e990d721e40-plugins-config-volume\") pod \"22e1eaae-3bfa-4163-b671-6e990d721e40\" (UID: \"22e1eaae-3bfa-4163-b671-6e990d721e40\") " Sep 07 16:45:10.666304 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:10.666280 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22e1eaae-3bfa-4163-b671-6e990d721e40-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "22e1eaae-3bfa-4163-b671-6e990d721e40" (UID: "22e1eaae-3bfa-4163-b671-6e990d721e40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 16:45:10.668322 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:10.668292 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22e1eaae-3bfa-4163-b671-6e990d721e40-kube-api-access-bl5vp" (OuterVolumeSpecName: "kube-api-access-bl5vp") pod "22e1eaae-3bfa-4163-b671-6e990d721e40" (UID: "22e1eaae-3bfa-4163-b671-6e990d721e40"). InnerVolumeSpecName "kube-api-access-bl5vp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 16:45:10.713187 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:10.713149 2580 generic.go:356] "Generic (PLEG): container finished" podID="22e1eaae-3bfa-4163-b671-6e990d721e40" containerID="b36045470286d910242b0c6c272fb479990e06a27f7d8e7cca5e6fbf71592039" exitCode=0 Sep 07 16:45:10.713345 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:10.713201 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4180e8-844b4ccfb6-xkcn5" Sep 07 16:45:10.713345 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:10.713245 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4180e8-844b4ccfb6-xkcn5" event={"ID":"22e1eaae-3bfa-4163-b671-6e990d721e40","Type":"ContainerDied","Data":"b36045470286d910242b0c6c272fb479990e06a27f7d8e7cca5e6fbf71592039"} Sep 07 16:45:10.713345 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:10.713303 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4180e8-844b4ccfb6-xkcn5" event={"ID":"22e1eaae-3bfa-4163-b671-6e990d721e40","Type":"ContainerDied","Data":"a83c247eed0ffd62a1f414a10adba5f28d9886024b4748979fd6fe482ba131db"} Sep 07 16:45:10.713345 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:10.713325 2580 scope.go:117] "RemoveContainer" containerID="b36045470286d910242b0c6c272fb479990e06a27f7d8e7cca5e6fbf71592039" Sep 07 16:45:10.722904 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:10.722886 2580 scope.go:117] "RemoveContainer" containerID="b36045470286d910242b0c6c272fb479990e06a27f7d8e7cca5e6fbf71592039" Sep 07 16:45:10.724578 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:45:10.724551 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b36045470286d910242b0c6c272fb479990e06a27f7d8e7cca5e6fbf71592039\": container with ID starting with b36045470286d910242b0c6c272fb479990e06a27f7d8e7cca5e6fbf71592039 not found: ID does not exist" containerID="b36045470286d910242b0c6c272fb479990e06a27f7d8e7cca5e6fbf71592039" Sep 07 16:45:10.724690 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:10.724589 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b36045470286d910242b0c6c272fb479990e06a27f7d8e7cca5e6fbf71592039"} err="failed to get container status \"b36045470286d910242b0c6c272fb479990e06a27f7d8e7cca5e6fbf71592039\": rpc error: code = NotFound desc = could not find container \"b36045470286d910242b0c6c272fb479990e06a27f7d8e7cca5e6fbf71592039\": container with ID starting with b36045470286d910242b0c6c272fb479990e06a27f7d8e7cca5e6fbf71592039 not found: ID does not exist" Sep 07 16:45:10.734452 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:10.734426 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4180e8-844b4ccfb6-xkcn5"] Sep 07 16:45:10.737977 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:10.737955 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4180e8-844b4ccfb6-xkcn5"] Sep 07 16:45:10.767390 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:10.767354 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22e1eaae-3bfa-4163-b671-6e990d721e40-plugins-config-volume\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:45:10.767390 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:10.767388 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl5vp\" (UniqueName: \"kubernetes.io/projected/22e1eaae-3bfa-4163-b671-6e990d721e40-kube-api-access-bl5vp\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:45:11.417737 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:11.417703 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0036d55a-4185-420c-9a0f-7c195bf806fc" path="/var/lib/kubelet/pods/0036d55a-4185-420c-9a0f-7c195bf806fc/volumes" Sep 07 16:45:11.418127 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:11.418085 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22e1eaae-3bfa-4163-b671-6e990d721e40" path="/var/lib/kubelet/pods/22e1eaae-3bfa-4163-b671-6e990d721e40/volumes" Sep 07 16:45:46.253049 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:46.252994 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf"] Sep 07 16:45:46.253490 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:46.253228 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf" podUID="f4331904-45b6-4e7f-9eec-cf23d280096d" containerName="istio-proxy" containerID="cri-o://358fea81adb1bfe6cdbe09bd61e48115731a8c47a60897b5f0ac572ea41c7b75" gracePeriod=30 Sep 07 16:45:51.513346 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:51.513317 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf" Sep 07 16:45:51.603795 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:51.603708 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x8gmp\" (UniqueName: \"kubernetes.io/projected/f4331904-45b6-4e7f-9eec-cf23d280096d-kube-api-access-x8gmp\") pod \"f4331904-45b6-4e7f-9eec-cf23d280096d\" (UID: \"f4331904-45b6-4e7f-9eec-cf23d280096d\") " Sep 07 16:45:51.603795 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:51.603756 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4331904-45b6-4e7f-9eec-cf23d280096d-istio-podinfo\") pod \"f4331904-45b6-4e7f-9eec-cf23d280096d\" (UID: \"f4331904-45b6-4e7f-9eec-cf23d280096d\") " Sep 07 16:45:51.603795 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:51.603790 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4331904-45b6-4e7f-9eec-cf23d280096d-istio-envoy\") pod \"f4331904-45b6-4e7f-9eec-cf23d280096d\" (UID: \"f4331904-45b6-4e7f-9eec-cf23d280096d\") " Sep 07 16:45:51.604005 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:51.603957 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4331904-45b6-4e7f-9eec-cf23d280096d-credential-socket\") pod \"f4331904-45b6-4e7f-9eec-cf23d280096d\" (UID: \"f4331904-45b6-4e7f-9eec-cf23d280096d\") " Sep 07 16:45:51.604088 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:51.604017 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4331904-45b6-4e7f-9eec-cf23d280096d-istiod-ca-cert\") pod \"f4331904-45b6-4e7f-9eec-cf23d280096d\" (UID: \"f4331904-45b6-4e7f-9eec-cf23d280096d\") " Sep 07 16:45:51.604146 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:51.604135 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4331904-45b6-4e7f-9eec-cf23d280096d-workload-socket\") pod \"f4331904-45b6-4e7f-9eec-cf23d280096d\" (UID: \"f4331904-45b6-4e7f-9eec-cf23d280096d\") " Sep 07 16:45:51.604201 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:51.604169 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4331904-45b6-4e7f-9eec-cf23d280096d-istio-token\") pod \"f4331904-45b6-4e7f-9eec-cf23d280096d\" (UID: \"f4331904-45b6-4e7f-9eec-cf23d280096d\") " Sep 07 16:45:51.604256 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:51.604222 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4331904-45b6-4e7f-9eec-cf23d280096d-workload-certs\") pod \"f4331904-45b6-4e7f-9eec-cf23d280096d\" (UID: \"f4331904-45b6-4e7f-9eec-cf23d280096d\") " Sep 07 16:45:51.604256 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:51.604243 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4331904-45b6-4e7f-9eec-cf23d280096d-istio-data\") pod \"f4331904-45b6-4e7f-9eec-cf23d280096d\" (UID: \"f4331904-45b6-4e7f-9eec-cf23d280096d\") " Sep 07 16:45:51.604356 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:51.604284 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4331904-45b6-4e7f-9eec-cf23d280096d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f4331904-45b6-4e7f-9eec-cf23d280096d" (UID: "f4331904-45b6-4e7f-9eec-cf23d280096d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 16:45:51.604723 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:51.604429 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4331904-45b6-4e7f-9eec-cf23d280096d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f4331904-45b6-4e7f-9eec-cf23d280096d" (UID: "f4331904-45b6-4e7f-9eec-cf23d280096d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 16:45:51.604723 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:51.604443 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4331904-45b6-4e7f-9eec-cf23d280096d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f4331904-45b6-4e7f-9eec-cf23d280096d" (UID: "f4331904-45b6-4e7f-9eec-cf23d280096d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 16:45:51.604723 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:51.604584 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4331904-45b6-4e7f-9eec-cf23d280096d-credential-socket\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:45:51.604723 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:51.604599 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4331904-45b6-4e7f-9eec-cf23d280096d-istiod-ca-cert\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:45:51.604723 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:51.604609 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4331904-45b6-4e7f-9eec-cf23d280096d-workload-socket\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:45:51.604723 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:51.604623 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4331904-45b6-4e7f-9eec-cf23d280096d-istio-data" (OuterVolumeSpecName: "istio-data") pod "f4331904-45b6-4e7f-9eec-cf23d280096d" (UID: "f4331904-45b6-4e7f-9eec-cf23d280096d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 16:45:51.605012 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:51.604746 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4331904-45b6-4e7f-9eec-cf23d280096d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f4331904-45b6-4e7f-9eec-cf23d280096d" (UID: "f4331904-45b6-4e7f-9eec-cf23d280096d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 16:45:51.606069 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:51.606039 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4331904-45b6-4e7f-9eec-cf23d280096d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f4331904-45b6-4e7f-9eec-cf23d280096d" (UID: "f4331904-45b6-4e7f-9eec-cf23d280096d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 16:45:51.606587 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:51.606553 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4331904-45b6-4e7f-9eec-cf23d280096d-istio-token" (OuterVolumeSpecName: "istio-token") pod "f4331904-45b6-4e7f-9eec-cf23d280096d" (UID: "f4331904-45b6-4e7f-9eec-cf23d280096d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 16:45:51.606670 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:51.606622 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4331904-45b6-4e7f-9eec-cf23d280096d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f4331904-45b6-4e7f-9eec-cf23d280096d" (UID: "f4331904-45b6-4e7f-9eec-cf23d280096d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 07 16:45:51.606670 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:51.606625 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4331904-45b6-4e7f-9eec-cf23d280096d-kube-api-access-x8gmp" (OuterVolumeSpecName: "kube-api-access-x8gmp") pod "f4331904-45b6-4e7f-9eec-cf23d280096d" (UID: "f4331904-45b6-4e7f-9eec-cf23d280096d"). InnerVolumeSpecName "kube-api-access-x8gmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 16:45:51.705041 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:51.704989 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4331904-45b6-4e7f-9eec-cf23d280096d-istio-token\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:45:51.705041 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:51.705015 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4331904-45b6-4e7f-9eec-cf23d280096d-workload-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:45:51.705041 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:51.705047 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4331904-45b6-4e7f-9eec-cf23d280096d-istio-data\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:45:51.705259 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:51.705056 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8gmp\" (UniqueName: \"kubernetes.io/projected/f4331904-45b6-4e7f-9eec-cf23d280096d-kube-api-access-x8gmp\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:45:51.705259 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:51.705066 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4331904-45b6-4e7f-9eec-cf23d280096d-istio-podinfo\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:45:51.705259 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:51.705074 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4331904-45b6-4e7f-9eec-cf23d280096d-istio-envoy\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:45:51.919135 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:51.919043 2580 generic.go:356] "Generic (PLEG): container finished" podID="f4331904-45b6-4e7f-9eec-cf23d280096d" containerID="358fea81adb1bfe6cdbe09bd61e48115731a8c47a60897b5f0ac572ea41c7b75" exitCode=0 Sep 07 16:45:51.919135 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:51.919073 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf" event={"ID":"f4331904-45b6-4e7f-9eec-cf23d280096d","Type":"ContainerDied","Data":"358fea81adb1bfe6cdbe09bd61e48115731a8c47a60897b5f0ac572ea41c7b75"} Sep 07 16:45:51.919135 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:51.919095 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf" Sep 07 16:45:51.919135 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:51.919133 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf" event={"ID":"f4331904-45b6-4e7f-9eec-cf23d280096d","Type":"ContainerDied","Data":"9b64b712bdb6a4fdc4abe9bafb161095e24e0d43d5d0c13024e4e25107af1b5f"} Sep 07 16:45:51.919405 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:51.919156 2580 scope.go:117] "RemoveContainer" containerID="358fea81adb1bfe6cdbe09bd61e48115731a8c47a60897b5f0ac572ea41c7b75" Sep 07 16:45:51.928707 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:51.928689 2580 scope.go:117] "RemoveContainer" containerID="6e59e425111da237dc999cb250f43174ec3b836723186cff581b457c65ceabe1" Sep 07 16:45:51.936864 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:51.936846 2580 scope.go:117] "RemoveContainer" containerID="358fea81adb1bfe6cdbe09bd61e48115731a8c47a60897b5f0ac572ea41c7b75" Sep 07 16:45:51.937168 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:45:51.937145 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"358fea81adb1bfe6cdbe09bd61e48115731a8c47a60897b5f0ac572ea41c7b75\": container with ID starting with 358fea81adb1bfe6cdbe09bd61e48115731a8c47a60897b5f0ac572ea41c7b75 not found: ID does not exist" containerID="358fea81adb1bfe6cdbe09bd61e48115731a8c47a60897b5f0ac572ea41c7b75" Sep 07 16:45:51.937236 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:51.937182 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"358fea81adb1bfe6cdbe09bd61e48115731a8c47a60897b5f0ac572ea41c7b75"} err="failed to get container status \"358fea81adb1bfe6cdbe09bd61e48115731a8c47a60897b5f0ac572ea41c7b75\": rpc error: code = NotFound desc = could not find container \"358fea81adb1bfe6cdbe09bd61e48115731a8c47a60897b5f0ac572ea41c7b75\": container with ID starting with 358fea81adb1bfe6cdbe09bd61e48115731a8c47a60897b5f0ac572ea41c7b75 not found: ID does not exist" Sep 07 16:45:51.937236 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:51.937207 2580 scope.go:117] "RemoveContainer" containerID="6e59e425111da237dc999cb250f43174ec3b836723186cff581b457c65ceabe1" Sep 07 16:45:51.937499 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:45:51.937482 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e59e425111da237dc999cb250f43174ec3b836723186cff581b457c65ceabe1\": container with ID starting with 6e59e425111da237dc999cb250f43174ec3b836723186cff581b457c65ceabe1 not found: ID does not exist" containerID="6e59e425111da237dc999cb250f43174ec3b836723186cff581b457c65ceabe1" Sep 07 16:45:51.937541 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:51.937506 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e59e425111da237dc999cb250f43174ec3b836723186cff581b457c65ceabe1"} err="failed to get container status \"6e59e425111da237dc999cb250f43174ec3b836723186cff581b457c65ceabe1\": rpc error: code = NotFound desc = could not find container \"6e59e425111da237dc999cb250f43174ec3b836723186cff581b457c65ceabe1\": container with ID starting with 6e59e425111da237dc999cb250f43174ec3b836723186cff581b457c65ceabe1 not found: ID does not exist" Sep 07 16:45:51.941241 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:51.941218 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf"] Sep 07 16:45:51.945086 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:51.945064 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-4180e8-openshift-default-76fddbf754-vkhnf"] Sep 07 16:45:53.418136 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:53.418103 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4331904-45b6-4e7f-9eec-cf23d280096d" path="/var/lib/kubelet/pods/f4331904-45b6-4e7f-9eec-cf23d280096d/volumes" Sep 07 16:45:57.072072 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:57.072040 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6a45e8-29813325-r5qbj"] Sep 07 16:45:57.075869 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:57.075840 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6a45e8-29813325-r5qbj"] Sep 07 16:45:57.097914 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:57.097873 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6a45e8-77b64f4b9c-v6tdm"] Sep 07 16:45:57.098243 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:57.098211 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a45e8-77b64f4b9c-v6tdm" podUID="0a837a0b-bcaf-4f08-ac76-37ec581a8ee0" containerName="payload-processing" containerID="cri-o://785bdfef3b128baa50849aa813b81af176fc8120e18e3a09a741a65c024b06dc" gracePeriod=30 Sep 07 16:45:57.355227 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:57.355180 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a45e8-77b64f4b9c-v6tdm" Sep 07 16:45:57.418707 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:57.418672 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02acb81e-f655-48b7-9163-010a2891ce7d" path="/var/lib/kubelet/pods/02acb81e-f655-48b7-9163-010a2891ce7d/volumes" Sep 07 16:45:57.457605 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:57.457564 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a837a0b-bcaf-4f08-ac76-37ec581a8ee0-plugins-config-volume\") pod \"0a837a0b-bcaf-4f08-ac76-37ec581a8ee0\" (UID: \"0a837a0b-bcaf-4f08-ac76-37ec581a8ee0\") " Sep 07 16:45:57.457782 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:57.457650 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bz7rr\" (UniqueName: \"kubernetes.io/projected/0a837a0b-bcaf-4f08-ac76-37ec581a8ee0-kube-api-access-bz7rr\") pod \"0a837a0b-bcaf-4f08-ac76-37ec581a8ee0\" (UID: \"0a837a0b-bcaf-4f08-ac76-37ec581a8ee0\") " Sep 07 16:45:57.457965 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:57.457939 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a837a0b-bcaf-4f08-ac76-37ec581a8ee0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0a837a0b-bcaf-4f08-ac76-37ec581a8ee0" (UID: "0a837a0b-bcaf-4f08-ac76-37ec581a8ee0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 16:45:57.460047 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:57.459990 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a837a0b-bcaf-4f08-ac76-37ec581a8ee0-kube-api-access-bz7rr" (OuterVolumeSpecName: "kube-api-access-bz7rr") pod "0a837a0b-bcaf-4f08-ac76-37ec581a8ee0" (UID: "0a837a0b-bcaf-4f08-ac76-37ec581a8ee0"). InnerVolumeSpecName "kube-api-access-bz7rr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 16:45:57.559245 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:57.559208 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bz7rr\" (UniqueName: \"kubernetes.io/projected/0a837a0b-bcaf-4f08-ac76-37ec581a8ee0-kube-api-access-bz7rr\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:45:57.559245 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:57.559238 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a837a0b-bcaf-4f08-ac76-37ec581a8ee0-plugins-config-volume\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:45:57.948953 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:57.948916 2580 generic.go:356] "Generic (PLEG): container finished" podID="0a837a0b-bcaf-4f08-ac76-37ec581a8ee0" containerID="785bdfef3b128baa50849aa813b81af176fc8120e18e3a09a741a65c024b06dc" exitCode=0 Sep 07 16:45:57.949161 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:57.948963 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a45e8-77b64f4b9c-v6tdm" Sep 07 16:45:57.949161 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:57.948976 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a45e8-77b64f4b9c-v6tdm" event={"ID":"0a837a0b-bcaf-4f08-ac76-37ec581a8ee0","Type":"ContainerDied","Data":"785bdfef3b128baa50849aa813b81af176fc8120e18e3a09a741a65c024b06dc"} Sep 07 16:45:57.949161 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:57.949017 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a45e8-77b64f4b9c-v6tdm" event={"ID":"0a837a0b-bcaf-4f08-ac76-37ec581a8ee0","Type":"ContainerDied","Data":"d876a0fd5882a398e383d580bc365593a7f8ded0cfe56086527ef0b8ce2c49bc"} Sep 07 16:45:57.949161 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:57.949062 2580 scope.go:117] "RemoveContainer" containerID="785bdfef3b128baa50849aa813b81af176fc8120e18e3a09a741a65c024b06dc" Sep 07 16:45:57.958232 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:57.958214 2580 scope.go:117] "RemoveContainer" containerID="785bdfef3b128baa50849aa813b81af176fc8120e18e3a09a741a65c024b06dc" Sep 07 16:45:57.958553 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:45:57.958534 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"785bdfef3b128baa50849aa813b81af176fc8120e18e3a09a741a65c024b06dc\": container with ID starting with 785bdfef3b128baa50849aa813b81af176fc8120e18e3a09a741a65c024b06dc not found: ID does not exist" containerID="785bdfef3b128baa50849aa813b81af176fc8120e18e3a09a741a65c024b06dc" Sep 07 16:45:57.958606 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:57.958562 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"785bdfef3b128baa50849aa813b81af176fc8120e18e3a09a741a65c024b06dc"} err="failed to get container status \"785bdfef3b128baa50849aa813b81af176fc8120e18e3a09a741a65c024b06dc\": rpc error: code = NotFound desc = could not find container \"785bdfef3b128baa50849aa813b81af176fc8120e18e3a09a741a65c024b06dc\": container with ID starting with 785bdfef3b128baa50849aa813b81af176fc8120e18e3a09a741a65c024b06dc not found: ID does not exist" Sep 07 16:45:57.969637 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:57.969609 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6a45e8-77b64f4b9c-v6tdm"] Sep 07 16:45:57.972986 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:57.972963 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6a45e8-77b64f4b9c-v6tdm"] Sep 07 16:45:59.417726 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:45:59.417695 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a837a0b-bcaf-4f08-ac76-37ec581a8ee0" path="/var/lib/kubelet/pods/0a837a0b-bcaf-4f08-ac76-37ec581a8ee0/volumes" Sep 07 16:46:04.047334 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:04.047304 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8d59e2-vh4nh"] Sep 07 16:46:04.047838 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:04.047623 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a837a0b-bcaf-4f08-ac76-37ec581a8ee0" containerName="payload-processing" Sep 07 16:46:04.047838 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:04.047634 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a837a0b-bcaf-4f08-ac76-37ec581a8ee0" containerName="payload-processing" Sep 07 16:46:04.047838 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:04.047642 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02acb81e-f655-48b7-9163-010a2891ce7d" containerName="cleanup" Sep 07 16:46:04.047838 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:04.047647 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="02acb81e-f655-48b7-9163-010a2891ce7d" containerName="cleanup" Sep 07 16:46:04.047838 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:04.047657 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4331904-45b6-4e7f-9eec-cf23d280096d" containerName="istio-proxy" Sep 07 16:46:04.047838 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:04.047663 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4331904-45b6-4e7f-9eec-cf23d280096d" containerName="istio-proxy" Sep 07 16:46:04.047838 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:04.047673 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22e1eaae-3bfa-4163-b671-6e990d721e40" containerName="payload-pre-processing" Sep 07 16:46:04.047838 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:04.047678 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="22e1eaae-3bfa-4163-b671-6e990d721e40" containerName="payload-pre-processing" Sep 07 16:46:04.047838 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:04.047690 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e8d6618-66bd-40c5-981a-b3f5667f9ef2" containerName="cleanup" Sep 07 16:46:04.047838 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:04.047695 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e8d6618-66bd-40c5-981a-b3f5667f9ef2" containerName="cleanup" Sep 07 16:46:04.047838 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:04.047702 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbcf587a-f5c9-4479-917d-1055a0eb395a" containerName="cleanup" Sep 07 16:46:04.047838 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:04.047706 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbcf587a-f5c9-4479-917d-1055a0eb395a" containerName="cleanup" Sep 07 16:46:04.047838 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:04.047754 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="02acb81e-f655-48b7-9163-010a2891ce7d" containerName="cleanup" Sep 07 16:46:04.047838 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:04.047761 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e8d6618-66bd-40c5-981a-b3f5667f9ef2" containerName="cleanup" Sep 07 16:46:04.047838 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:04.047767 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a837a0b-bcaf-4f08-ac76-37ec581a8ee0" containerName="payload-processing" Sep 07 16:46:04.047838 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:04.047772 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbcf587a-f5c9-4479-917d-1055a0eb395a" containerName="cleanup" Sep 07 16:46:04.047838 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:04.047778 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4331904-45b6-4e7f-9eec-cf23d280096d" containerName="istio-proxy" Sep 07 16:46:04.047838 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:04.047783 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4331904-45b6-4e7f-9eec-cf23d280096d" containerName="istio-proxy" Sep 07 16:46:04.047838 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:04.047789 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="22e1eaae-3bfa-4163-b671-6e990d721e40" containerName="payload-pre-processing" Sep 07 16:46:04.050880 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:04.050862 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8d59e2-vh4nh" Sep 07 16:46:04.053141 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:04.053118 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 07 16:46:04.053272 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:04.053118 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jpjxt\"" Sep 07 16:46:04.058726 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:04.058696 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8d59e2-vh4nh"] Sep 07 16:46:04.116333 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:04.116301 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ae97e76-b802-4471-bf19-089a589714d0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-8d59e2-vh4nh\" (UID: \"6ae97e76-b802-4471-bf19-089a589714d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8d59e2-vh4nh" Sep 07 16:46:04.217621 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:04.217586 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ae97e76-b802-4471-bf19-089a589714d0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-8d59e2-vh4nh\" (UID: \"6ae97e76-b802-4471-bf19-089a589714d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8d59e2-vh4nh" Sep 07 16:46:04.218206 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:04.218188 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ae97e76-b802-4471-bf19-089a589714d0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-8d59e2-vh4nh\" (UID: \"6ae97e76-b802-4471-bf19-089a589714d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8d59e2-vh4nh" Sep 07 16:46:04.362000 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:04.361909 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8d59e2-vh4nh" Sep 07 16:46:04.653213 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:04.653127 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8d59e2-vh4nh"] Sep 07 16:46:04.656168 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:46:04.656142 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ae97e76_b802_4471_bf19_089a589714d0.slice/crio-4f2190ab06ca14fe8ec9163fd48dad27f956e09781dbe9b3a753182aeff0533c WatchSource:0}: Error finding container 4f2190ab06ca14fe8ec9163fd48dad27f956e09781dbe9b3a753182aeff0533c: Status 404 returned error can't find the container with id 4f2190ab06ca14fe8ec9163fd48dad27f956e09781dbe9b3a753182aeff0533c Sep 07 16:46:04.984280 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:04.984243 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8d59e2-vh4nh" event={"ID":"6ae97e76-b802-4471-bf19-089a589714d0","Type":"ContainerStarted","Data":"f88b696a92be0f6b1d4492005ae20a959231015aa4b62fc80eecb108e755f0ee"} Sep 07 16:46:04.984451 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:04.984297 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8d59e2-vh4nh" event={"ID":"6ae97e76-b802-4471-bf19-089a589714d0","Type":"ContainerStarted","Data":"4f2190ab06ca14fe8ec9163fd48dad27f956e09781dbe9b3a753182aeff0533c"} Sep 07 16:46:04.998740 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:04.998676 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8d59e2-vh4nh" podStartSLOduration=0.998657354 podStartE2EDuration="998.657354ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 16:46:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:46:04.998607713 +0000 UTC m=+1594.033830785" watchObservedRunningTime="2026-09-07 16:46:04.998657354 +0000 UTC m=+1594.033880426" Sep 07 16:46:05.991075 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:05.991044 2580 generic.go:356] "Generic (PLEG): container finished" podID="6ae97e76-b802-4471-bf19-089a589714d0" containerID="f88b696a92be0f6b1d4492005ae20a959231015aa4b62fc80eecb108e755f0ee" exitCode=0 Sep 07 16:46:05.991552 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:05.991118 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8d59e2-vh4nh" event={"ID":"6ae97e76-b802-4471-bf19-089a589714d0","Type":"ContainerDied","Data":"f88b696a92be0f6b1d4492005ae20a959231015aa4b62fc80eecb108e755f0ee"} Sep 07 16:46:07.123493 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:07.123464 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8d59e2-vh4nh" Sep 07 16:46:07.245086 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:07.245042 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ae97e76-b802-4471-bf19-089a589714d0-maas-api-service-ca\") pod \"6ae97e76-b802-4471-bf19-089a589714d0\" (UID: \"6ae97e76-b802-4471-bf19-089a589714d0\") " Sep 07 16:46:07.245455 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:07.245436 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ae97e76-b802-4471-bf19-089a589714d0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6ae97e76-b802-4471-bf19-089a589714d0" (UID: "6ae97e76-b802-4471-bf19-089a589714d0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 16:46:07.345825 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:07.345756 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ae97e76-b802-4471-bf19-089a589714d0-maas-api-service-ca\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:46:08.001665 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:08.001634 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8d59e2-vh4nh" Sep 07 16:46:08.001830 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:08.001628 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8d59e2-vh4nh" event={"ID":"6ae97e76-b802-4471-bf19-089a589714d0","Type":"ContainerDied","Data":"4f2190ab06ca14fe8ec9163fd48dad27f956e09781dbe9b3a753182aeff0533c"} Sep 07 16:46:08.001830 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:08.001760 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f2190ab06ca14fe8ec9163fd48dad27f956e09781dbe9b3a753182aeff0533c" Sep 07 16:46:14.702739 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:14.702704 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8d59e2-29813325-99pb7"] Sep 07 16:46:14.704496 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:14.704472 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8d59e2-29813325-99pb7"] Sep 07 16:46:14.906189 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:14.906149 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8d59e2-vh4nh"] Sep 07 16:46:14.908176 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:14.908151 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8d59e2-vh4nh"] Sep 07 16:46:15.417732 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:15.417699 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e8d6618-66bd-40c5-981a-b3f5667f9ef2" path="/var/lib/kubelet/pods/1e8d6618-66bd-40c5-981a-b3f5667f9ef2/volumes" Sep 07 16:46:15.418031 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:15.418007 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ae97e76-b802-4471-bf19-089a589714d0" path="/var/lib/kubelet/pods/6ae97e76-b802-4471-bf19-089a589714d0/volumes" Sep 07 16:46:32.292171 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:32.292132 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7aca0f-29813325-knggl"] Sep 07 16:46:32.295518 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:32.295480 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7aca0f-29813325-knggl"] Sep 07 16:46:32.326886 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:32.326852 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7aca0f-79db6b5986-m7dgg"] Sep 07 16:46:32.327379 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:32.327347 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7aca0f-79db6b5986-m7dgg" podUID="1ae56c63-f8be-4d84-ad66-8503d5ca3e0e" containerName="payload-pre-processing" containerID="cri-o://f566236a5a8f88af44eb63952a0a7f05c71dc472f12eea54f08fbbda1269afb4" gracePeriod=30 Sep 07 16:46:32.586276 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:32.586255 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7aca0f-79db6b5986-m7dgg" Sep 07 16:46:32.681091 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:32.681054 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-krkkc\" (UniqueName: \"kubernetes.io/projected/1ae56c63-f8be-4d84-ad66-8503d5ca3e0e-kube-api-access-krkkc\") pod \"1ae56c63-f8be-4d84-ad66-8503d5ca3e0e\" (UID: \"1ae56c63-f8be-4d84-ad66-8503d5ca3e0e\") " Sep 07 16:46:32.681288 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:32.681172 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae56c63-f8be-4d84-ad66-8503d5ca3e0e-plugins-config-volume\") pod \"1ae56c63-f8be-4d84-ad66-8503d5ca3e0e\" (UID: \"1ae56c63-f8be-4d84-ad66-8503d5ca3e0e\") " Sep 07 16:46:32.681666 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:32.681619 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ae56c63-f8be-4d84-ad66-8503d5ca3e0e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1ae56c63-f8be-4d84-ad66-8503d5ca3e0e" (UID: "1ae56c63-f8be-4d84-ad66-8503d5ca3e0e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 16:46:32.683824 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:32.683780 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ae56c63-f8be-4d84-ad66-8503d5ca3e0e-kube-api-access-krkkc" (OuterVolumeSpecName: "kube-api-access-krkkc") pod "1ae56c63-f8be-4d84-ad66-8503d5ca3e0e" (UID: "1ae56c63-f8be-4d84-ad66-8503d5ca3e0e"). InnerVolumeSpecName "kube-api-access-krkkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 16:46:32.782647 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:32.782582 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae56c63-f8be-4d84-ad66-8503d5ca3e0e-plugins-config-volume\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:46:32.782647 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:32.782638 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krkkc\" (UniqueName: \"kubernetes.io/projected/1ae56c63-f8be-4d84-ad66-8503d5ca3e0e-kube-api-access-krkkc\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:46:33.127911 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:33.127874 2580 generic.go:356] "Generic (PLEG): container finished" podID="1ae56c63-f8be-4d84-ad66-8503d5ca3e0e" containerID="f566236a5a8f88af44eb63952a0a7f05c71dc472f12eea54f08fbbda1269afb4" exitCode=0 Sep 07 16:46:33.128109 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:33.127919 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7aca0f-79db6b5986-m7dgg" Sep 07 16:46:33.128109 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:33.127934 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7aca0f-79db6b5986-m7dgg" event={"ID":"1ae56c63-f8be-4d84-ad66-8503d5ca3e0e","Type":"ContainerDied","Data":"f566236a5a8f88af44eb63952a0a7f05c71dc472f12eea54f08fbbda1269afb4"} Sep 07 16:46:33.128109 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:33.127987 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7aca0f-79db6b5986-m7dgg" event={"ID":"1ae56c63-f8be-4d84-ad66-8503d5ca3e0e","Type":"ContainerDied","Data":"5e66558030f0d0e52ec299f06b5af0ba4534e405df74b699a65f9dc96bdacd42"} Sep 07 16:46:33.128109 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:33.128017 2580 scope.go:117] "RemoveContainer" containerID="f566236a5a8f88af44eb63952a0a7f05c71dc472f12eea54f08fbbda1269afb4" Sep 07 16:46:33.136902 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:33.136884 2580 scope.go:117] "RemoveContainer" containerID="f566236a5a8f88af44eb63952a0a7f05c71dc472f12eea54f08fbbda1269afb4" Sep 07 16:46:33.142784 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:46:33.142748 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f566236a5a8f88af44eb63952a0a7f05c71dc472f12eea54f08fbbda1269afb4\": container with ID starting with f566236a5a8f88af44eb63952a0a7f05c71dc472f12eea54f08fbbda1269afb4 not found: ID does not exist" containerID="f566236a5a8f88af44eb63952a0a7f05c71dc472f12eea54f08fbbda1269afb4" Sep 07 16:46:33.142903 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:33.142794 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f566236a5a8f88af44eb63952a0a7f05c71dc472f12eea54f08fbbda1269afb4"} err="failed to get container status \"f566236a5a8f88af44eb63952a0a7f05c71dc472f12eea54f08fbbda1269afb4\": rpc error: code = NotFound desc = could not find container \"f566236a5a8f88af44eb63952a0a7f05c71dc472f12eea54f08fbbda1269afb4\": container with ID starting with f566236a5a8f88af44eb63952a0a7f05c71dc472f12eea54f08fbbda1269afb4 not found: ID does not exist" Sep 07 16:46:33.151996 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:33.151957 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7aca0f-79db6b5986-m7dgg"] Sep 07 16:46:33.157839 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:33.157817 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7aca0f-79db6b5986-m7dgg"] Sep 07 16:46:33.417656 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:33.417581 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ae56c63-f8be-4d84-ad66-8503d5ca3e0e" path="/var/lib/kubelet/pods/1ae56c63-f8be-4d84-ad66-8503d5ca3e0e/volumes" Sep 07 16:46:33.417984 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:33.417858 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbcf587a-f5c9-4479-917d-1055a0eb395a" path="/var/lib/kubelet/pods/bbcf587a-f5c9-4479-917d-1055a0eb395a/volumes" Sep 07 16:46:38.871052 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:38.870992 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n"] Sep 07 16:46:38.871641 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:38.871214 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n" podUID="7f620b12-a4d0-4b27-b702-be0f8c74e22e" containerName="istio-proxy" containerID="cri-o://1ff39bc4d94daf0681122a74ff8ce6844d4965eb2abe9fbb992b21206d2dab20" gracePeriod=30 Sep 07 16:46:44.116049 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:44.116010 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n" Sep 07 16:46:44.181237 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:44.181143 2580 generic.go:356] "Generic (PLEG): container finished" podID="7f620b12-a4d0-4b27-b702-be0f8c74e22e" containerID="1ff39bc4d94daf0681122a74ff8ce6844d4965eb2abe9fbb992b21206d2dab20" exitCode=0 Sep 07 16:46:44.181237 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:44.181193 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n" Sep 07 16:46:44.181237 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:44.181226 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n" event={"ID":"7f620b12-a4d0-4b27-b702-be0f8c74e22e","Type":"ContainerDied","Data":"1ff39bc4d94daf0681122a74ff8ce6844d4965eb2abe9fbb992b21206d2dab20"} Sep 07 16:46:44.181512 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:44.181286 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n" event={"ID":"7f620b12-a4d0-4b27-b702-be0f8c74e22e","Type":"ContainerDied","Data":"787879d7d5b343e40eb2d1c4e55518cfe508f091ec0e7ce740e671c79ce56266"} Sep 07 16:46:44.181512 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:44.181310 2580 scope.go:117] "RemoveContainer" containerID="1ff39bc4d94daf0681122a74ff8ce6844d4965eb2abe9fbb992b21206d2dab20" Sep 07 16:46:44.181989 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:44.181962 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f620b12-a4d0-4b27-b702-be0f8c74e22e-istio-podinfo\") pod \"7f620b12-a4d0-4b27-b702-be0f8c74e22e\" (UID: \"7f620b12-a4d0-4b27-b702-be0f8c74e22e\") " Sep 07 16:46:44.182144 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:44.182071 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f620b12-a4d0-4b27-b702-be0f8c74e22e-credential-socket\") pod \"7f620b12-a4d0-4b27-b702-be0f8c74e22e\" (UID: \"7f620b12-a4d0-4b27-b702-be0f8c74e22e\") " Sep 07 16:46:44.182144 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:44.182096 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f620b12-a4d0-4b27-b702-be0f8c74e22e-istio-token\") pod \"7f620b12-a4d0-4b27-b702-be0f8c74e22e\" (UID: \"7f620b12-a4d0-4b27-b702-be0f8c74e22e\") " Sep 07 16:46:44.182144 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:44.182124 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f620b12-a4d0-4b27-b702-be0f8c74e22e-istio-envoy\") pod \"7f620b12-a4d0-4b27-b702-be0f8c74e22e\" (UID: \"7f620b12-a4d0-4b27-b702-be0f8c74e22e\") " Sep 07 16:46:44.182319 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:44.182147 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-26l88\" (UniqueName: \"kubernetes.io/projected/7f620b12-a4d0-4b27-b702-be0f8c74e22e-kube-api-access-26l88\") pod \"7f620b12-a4d0-4b27-b702-be0f8c74e22e\" (UID: \"7f620b12-a4d0-4b27-b702-be0f8c74e22e\") " Sep 07 16:46:44.182319 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:44.182169 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f620b12-a4d0-4b27-b702-be0f8c74e22e-istio-data\") pod \"7f620b12-a4d0-4b27-b702-be0f8c74e22e\" (UID: \"7f620b12-a4d0-4b27-b702-be0f8c74e22e\") " Sep 07 16:46:44.182319 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:44.182192 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f620b12-a4d0-4b27-b702-be0f8c74e22e-workload-socket\") pod \"7f620b12-a4d0-4b27-b702-be0f8c74e22e\" (UID: \"7f620b12-a4d0-4b27-b702-be0f8c74e22e\") " Sep 07 16:46:44.182319 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:44.182223 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f620b12-a4d0-4b27-b702-be0f8c74e22e-istiod-ca-cert\") pod \"7f620b12-a4d0-4b27-b702-be0f8c74e22e\" (UID: \"7f620b12-a4d0-4b27-b702-be0f8c74e22e\") " Sep 07 16:46:44.182319 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:44.182283 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f620b12-a4d0-4b27-b702-be0f8c74e22e-workload-certs\") pod \"7f620b12-a4d0-4b27-b702-be0f8c74e22e\" (UID: \"7f620b12-a4d0-4b27-b702-be0f8c74e22e\") " Sep 07 16:46:44.182678 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:44.182338 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f620b12-a4d0-4b27-b702-be0f8c74e22e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7f620b12-a4d0-4b27-b702-be0f8c74e22e" (UID: "7f620b12-a4d0-4b27-b702-be0f8c74e22e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 16:46:44.182678 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:44.182607 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f620b12-a4d0-4b27-b702-be0f8c74e22e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7f620b12-a4d0-4b27-b702-be0f8c74e22e" (UID: "7f620b12-a4d0-4b27-b702-be0f8c74e22e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 16:46:44.182794 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:44.182735 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f620b12-a4d0-4b27-b702-be0f8c74e22e-istio-data" (OuterVolumeSpecName: "istio-data") pod "7f620b12-a4d0-4b27-b702-be0f8c74e22e" (UID: "7f620b12-a4d0-4b27-b702-be0f8c74e22e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 16:46:44.182932 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:44.182888 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f620b12-a4d0-4b27-b702-be0f8c74e22e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7f620b12-a4d0-4b27-b702-be0f8c74e22e" (UID: "7f620b12-a4d0-4b27-b702-be0f8c74e22e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 16:46:44.183186 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:44.183166 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f620b12-a4d0-4b27-b702-be0f8c74e22e-istio-data\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:46:44.183271 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:44.183202 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f620b12-a4d0-4b27-b702-be0f8c74e22e-workload-socket\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:46:44.183271 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:44.183219 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f620b12-a4d0-4b27-b702-be0f8c74e22e-workload-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:46:44.183271 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:44.183233 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f620b12-a4d0-4b27-b702-be0f8c74e22e-credential-socket\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:46:44.183413 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:44.183283 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f620b12-a4d0-4b27-b702-be0f8c74e22e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7f620b12-a4d0-4b27-b702-be0f8c74e22e" (UID: "7f620b12-a4d0-4b27-b702-be0f8c74e22e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 16:46:44.184818 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:44.184769 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f620b12-a4d0-4b27-b702-be0f8c74e22e-kube-api-access-26l88" (OuterVolumeSpecName: "kube-api-access-26l88") pod "7f620b12-a4d0-4b27-b702-be0f8c74e22e" (UID: "7f620b12-a4d0-4b27-b702-be0f8c74e22e"). InnerVolumeSpecName "kube-api-access-26l88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 16:46:44.184915 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:44.184857 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f620b12-a4d0-4b27-b702-be0f8c74e22e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7f620b12-a4d0-4b27-b702-be0f8c74e22e" (UID: "7f620b12-a4d0-4b27-b702-be0f8c74e22e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 07 16:46:44.184915 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:44.184869 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f620b12-a4d0-4b27-b702-be0f8c74e22e-istio-token" (OuterVolumeSpecName: "istio-token") pod "7f620b12-a4d0-4b27-b702-be0f8c74e22e" (UID: "7f620b12-a4d0-4b27-b702-be0f8c74e22e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 16:46:44.185408 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:44.185389 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f620b12-a4d0-4b27-b702-be0f8c74e22e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7f620b12-a4d0-4b27-b702-be0f8c74e22e" (UID: "7f620b12-a4d0-4b27-b702-be0f8c74e22e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 16:46:44.201241 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:44.201218 2580 scope.go:117] "RemoveContainer" containerID="5df1cdc037c82bdb52ad2fbcdc47f42ac151332e9f5fd5ad0d680e864b15e87a" Sep 07 16:46:44.209922 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:44.209896 2580 scope.go:117] "RemoveContainer" containerID="1ff39bc4d94daf0681122a74ff8ce6844d4965eb2abe9fbb992b21206d2dab20" Sep 07 16:46:44.210279 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:46:44.210260 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ff39bc4d94daf0681122a74ff8ce6844d4965eb2abe9fbb992b21206d2dab20\": container with ID starting with 1ff39bc4d94daf0681122a74ff8ce6844d4965eb2abe9fbb992b21206d2dab20 not found: ID does not exist" containerID="1ff39bc4d94daf0681122a74ff8ce6844d4965eb2abe9fbb992b21206d2dab20" Sep 07 16:46:44.210351 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:44.210289 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ff39bc4d94daf0681122a74ff8ce6844d4965eb2abe9fbb992b21206d2dab20"} err="failed to get container status \"1ff39bc4d94daf0681122a74ff8ce6844d4965eb2abe9fbb992b21206d2dab20\": rpc error: code = NotFound desc = could not find container \"1ff39bc4d94daf0681122a74ff8ce6844d4965eb2abe9fbb992b21206d2dab20\": container with ID starting with 1ff39bc4d94daf0681122a74ff8ce6844d4965eb2abe9fbb992b21206d2dab20 not found: ID does not exist" Sep 07 16:46:44.210351 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:44.210310 2580 scope.go:117] "RemoveContainer" containerID="5df1cdc037c82bdb52ad2fbcdc47f42ac151332e9f5fd5ad0d680e864b15e87a" Sep 07 16:46:44.210600 ip-10-0-135-242 kubenswrapper[2580]: E0907 16:46:44.210584 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5df1cdc037c82bdb52ad2fbcdc47f42ac151332e9f5fd5ad0d680e864b15e87a\": container with ID starting with 5df1cdc037c82bdb52ad2fbcdc47f42ac151332e9f5fd5ad0d680e864b15e87a not found: ID does not exist" containerID="5df1cdc037c82bdb52ad2fbcdc47f42ac151332e9f5fd5ad0d680e864b15e87a" Sep 07 16:46:44.210642 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:44.210605 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5df1cdc037c82bdb52ad2fbcdc47f42ac151332e9f5fd5ad0d680e864b15e87a"} err="failed to get container status \"5df1cdc037c82bdb52ad2fbcdc47f42ac151332e9f5fd5ad0d680e864b15e87a\": rpc error: code = NotFound desc = could not find container \"5df1cdc037c82bdb52ad2fbcdc47f42ac151332e9f5fd5ad0d680e864b15e87a\": container with ID starting with 5df1cdc037c82bdb52ad2fbcdc47f42ac151332e9f5fd5ad0d680e864b15e87a not found: ID does not exist" Sep 07 16:46:44.284424 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:44.284382 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f620b12-a4d0-4b27-b702-be0f8c74e22e-istio-podinfo\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:46:44.284424 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:44.284415 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f620b12-a4d0-4b27-b702-be0f8c74e22e-istio-token\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:46:44.284424 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:44.284426 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f620b12-a4d0-4b27-b702-be0f8c74e22e-istio-envoy\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:46:44.284424 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:44.284434 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26l88\" (UniqueName: \"kubernetes.io/projected/7f620b12-a4d0-4b27-b702-be0f8c74e22e-kube-api-access-26l88\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:46:44.284694 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:44.284448 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f620b12-a4d0-4b27-b702-be0f8c74e22e-istiod-ca-cert\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\"" Sep 07 16:46:44.503631 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:44.503601 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n"] Sep 07 16:46:44.509377 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:44.509353 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7aca0f-openshift-default-67cdf9f46-j4z2n"] Sep 07 16:46:45.418208 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:46:45.418178 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f620b12-a4d0-4b27-b702-be0f8c74e22e" path="/var/lib/kubelet/pods/7f620b12-a4d0-4b27-b702-be0f8c74e22e/volumes" Sep 07 16:48:31.765521 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:48:31.765488 2580 scope.go:117] "RemoveContainer" containerID="6c27ccbe9f2458ad7380109a15bd3256705374f5fcb31fc49a340341a396a38b" Sep 07 16:48:31.773665 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:48:31.773646 2580 scope.go:117] "RemoveContainer" containerID="7e0ae02ce73e7818cdfe0caa55ea0a9dbae1a5ba3535cf11d8359d7517b9dfd3" Sep 07 16:48:31.783244 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:48:31.783224 2580 scope.go:117] "RemoveContainer" containerID="fdc043b2048faaa72363fd5c600fd134bb33e03d32d7207d527410b2c752cb31" Sep 07 16:51:31.818916 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:51:31.818882 2580 scope.go:117] "RemoveContainer" containerID="99902eb549123a8a11f3d0911452f8ede9314c51897dd4e2920c08d72cc5aff7" Sep 07 16:51:31.827039 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:51:31.827001 2580 scope.go:117] "RemoveContainer" containerID="40ba1106974e392e7ea12034d2a3e01e0f98028e2b4d2a1a36ae336b8690d957" Sep 07 16:51:31.834521 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:51:31.834499 2580 scope.go:117] "RemoveContainer" containerID="92c4c83723e8466ff7a66250fd524cea6c98dbc702872c03a362c326f712cccb" Sep 07 16:51:31.841629 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:51:31.841599 2580 scope.go:117] "RemoveContainer" containerID="f587351a838ef8c64a532a8f0886cf023eaf1579ed5a6ce3e151a1fb3dbc69c7" Sep 07 16:52:31.867633 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:52:31.867547 2580 scope.go:117] "RemoveContainer" containerID="f88b696a92be0f6b1d4492005ae20a959231015aa4b62fc80eecb108e755f0ee" Sep 07 16:52:44.370302 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:52:44.370264 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7766cc5489-hcs8n_8c454a42-b87a-4061-b104-522b821e3978/manager/0.log" Sep 07 16:52:44.376413 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:52:44.376394 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7766cc5489-hcs8n_8c454a42-b87a-4061-b104-522b821e3978/copy-manifests/0.log" Sep 07 16:52:46.894276 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:52:46.894244 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg_8fe982b2-5675-4a4b-b8a7-5cc0c397480c/istio-proxy/0.log" Sep 07 16:52:47.014210 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:52:47.014173 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-2rc48_182ffcc1-816e-4871-a357-6b9ac07e4481/discovery/0.log" Sep 07 16:52:47.446089 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:52:47.446052 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-wpflr_174d0f3e-5f11-4428-8839-784bff454949/payload-pre-processing/0.log" Sep 07 16:52:48.241830 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:52:48.241799 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-mmfjx_b11b65ed-c657-42bd-9d79-d641d429f300/main/0.log" Sep 07 16:52:48.353753 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:52:48.353724 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-rlkbs_a500950d-e0e6-4542-9f0a-b388e5f09b76/main/0.log" Sep 07 16:52:48.460802 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:52:48.460757 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjgstf_4897fdef-8bf5-485d-8e7b-51e239ad09f4/main/0.log" Sep 07 16:52:48.572316 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:52:48.572232 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-lxzg7_1c26734a-083f-468f-8f8d-a3ced9e3c898/main/0.log" Sep 07 16:52:56.292561 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:52:56.292531 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6n92f_4f4aaec6-7ab4-48ab-a0e1-4d26977f0315/global-pull-secret-syncer/0.log" Sep 07 16:52:56.537294 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:52:56.537263 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-hv5bm_4013a04f-2b59-44fa-af92-aece71a27fc8/konnectivity-agent/0.log" Sep 07 16:52:56.654413 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:52:56.654288 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-242.ec2.internal_c620202158ca74d37111884761e47e70/haproxy/0.log" Sep 07 16:53:03.769144 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:03.769105 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cpc9m_04187790-e6b1-4739-94d5-a7d20645ae1c/node-exporter/0.log" Sep 07 16:53:03.791140 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:03.791107 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cpc9m_04187790-e6b1-4739-94d5-a7d20645ae1c/kube-rbac-proxy/0.log" Sep 07 16:53:03.811947 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:03.811927 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cpc9m_04187790-e6b1-4739-94d5-a7d20645ae1c/init-textfile/0.log" Sep 07 16:53:06.226680 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:06.226639 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5v74r/perf-node-gather-daemonset-zbfxs"] Sep 07 16:53:06.227177 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:06.227096 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f620b12-a4d0-4b27-b702-be0f8c74e22e" containerName="istio-proxy" Sep 07 16:53:06.227177 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:06.227117 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f620b12-a4d0-4b27-b702-be0f8c74e22e" containerName="istio-proxy" Sep 07 16:53:06.227177 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:06.227133 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4331904-45b6-4e7f-9eec-cf23d280096d" containerName="istio-proxy" Sep 07 16:53:06.227177 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:06.227140 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4331904-45b6-4e7f-9eec-cf23d280096d" containerName="istio-proxy" Sep 07 16:53:06.227177 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:06.227153 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ae97e76-b802-4471-bf19-089a589714d0" containerName="revoke-keys" Sep 07 16:53:06.227177 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:06.227163 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ae97e76-b802-4471-bf19-089a589714d0" containerName="revoke-keys" Sep 07 16:53:06.227515 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:06.227195 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ae56c63-f8be-4d84-ad66-8503d5ca3e0e" containerName="payload-pre-processing" Sep 07 16:53:06.227515 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:06.227206 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ae56c63-f8be-4d84-ad66-8503d5ca3e0e" containerName="payload-pre-processing" Sep 07 16:53:06.227515 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:06.227283 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ae56c63-f8be-4d84-ad66-8503d5ca3e0e" containerName="payload-pre-processing" Sep 07 16:53:06.227515 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:06.227296 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f620b12-a4d0-4b27-b702-be0f8c74e22e" containerName="istio-proxy" Sep 07 16:53:06.227515 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:06.227307 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ae97e76-b802-4471-bf19-089a589714d0" containerName="revoke-keys" Sep 07 16:53:06.227515 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:06.227315 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f620b12-a4d0-4b27-b702-be0f8c74e22e" containerName="istio-proxy" Sep 07 16:53:06.230498 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:06.230476 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5v74r/perf-node-gather-daemonset-zbfxs" Sep 07 16:53:06.232532 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:06.232511 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5v74r\"/\"default-dockercfg-trgjt\"" Sep 07 16:53:06.232655 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:06.232514 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5v74r\"/\"kube-root-ca.crt\"" Sep 07 16:53:06.232655 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:06.232514 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5v74r\"/\"openshift-service-ca.crt\"" Sep 07 16:53:06.239674 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:06.239650 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5v74r/perf-node-gather-daemonset-zbfxs"] Sep 07 16:53:06.305793 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:06.305759 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/da82d61d-3311-4868-b781-ff2c489f3ffd-sys\") pod \"perf-node-gather-daemonset-zbfxs\" (UID: \"da82d61d-3311-4868-b781-ff2c489f3ffd\") " pod="openshift-must-gather-5v74r/perf-node-gather-daemonset-zbfxs" Sep 07 16:53:06.305977 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:06.305804 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/da82d61d-3311-4868-b781-ff2c489f3ffd-proc\") pod \"perf-node-gather-daemonset-zbfxs\" (UID: \"da82d61d-3311-4868-b781-ff2c489f3ffd\") " pod="openshift-must-gather-5v74r/perf-node-gather-daemonset-zbfxs" Sep 07 16:53:06.305977 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:06.305868 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/da82d61d-3311-4868-b781-ff2c489f3ffd-podres\") pod \"perf-node-gather-daemonset-zbfxs\" (UID: \"da82d61d-3311-4868-b781-ff2c489f3ffd\") " pod="openshift-must-gather-5v74r/perf-node-gather-daemonset-zbfxs" Sep 07 16:53:06.305977 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:06.305899 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t6wx\" (UniqueName: \"kubernetes.io/projected/da82d61d-3311-4868-b781-ff2c489f3ffd-kube-api-access-6t6wx\") pod \"perf-node-gather-daemonset-zbfxs\" (UID: \"da82d61d-3311-4868-b781-ff2c489f3ffd\") " pod="openshift-must-gather-5v74r/perf-node-gather-daemonset-zbfxs" Sep 07 16:53:06.305977 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:06.305919 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/da82d61d-3311-4868-b781-ff2c489f3ffd-lib-modules\") pod \"perf-node-gather-daemonset-zbfxs\" (UID: \"da82d61d-3311-4868-b781-ff2c489f3ffd\") " pod="openshift-must-gather-5v74r/perf-node-gather-daemonset-zbfxs" Sep 07 16:53:06.406382 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:06.406350 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/da82d61d-3311-4868-b781-ff2c489f3ffd-sys\") pod \"perf-node-gather-daemonset-zbfxs\" (UID: \"da82d61d-3311-4868-b781-ff2c489f3ffd\") " pod="openshift-must-gather-5v74r/perf-node-gather-daemonset-zbfxs" Sep 07 16:53:06.406382 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:06.406386 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/da82d61d-3311-4868-b781-ff2c489f3ffd-proc\") pod \"perf-node-gather-daemonset-zbfxs\" (UID: \"da82d61d-3311-4868-b781-ff2c489f3ffd\") " pod="openshift-must-gather-5v74r/perf-node-gather-daemonset-zbfxs" Sep 07 16:53:06.406591 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:06.406445 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/da82d61d-3311-4868-b781-ff2c489f3ffd-podres\") pod \"perf-node-gather-daemonset-zbfxs\" (UID: \"da82d61d-3311-4868-b781-ff2c489f3ffd\") " pod="openshift-must-gather-5v74r/perf-node-gather-daemonset-zbfxs" Sep 07 16:53:06.406591 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:06.406470 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t6wx\" (UniqueName: \"kubernetes.io/projected/da82d61d-3311-4868-b781-ff2c489f3ffd-kube-api-access-6t6wx\") pod \"perf-node-gather-daemonset-zbfxs\" (UID: \"da82d61d-3311-4868-b781-ff2c489f3ffd\") " pod="openshift-must-gather-5v74r/perf-node-gather-daemonset-zbfxs" Sep 07 16:53:06.406591 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:06.406477 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/da82d61d-3311-4868-b781-ff2c489f3ffd-sys\") pod \"perf-node-gather-daemonset-zbfxs\" (UID: \"da82d61d-3311-4868-b781-ff2c489f3ffd\") " pod="openshift-must-gather-5v74r/perf-node-gather-daemonset-zbfxs" Sep 07 16:53:06.406591 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:06.406497 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/da82d61d-3311-4868-b781-ff2c489f3ffd-lib-modules\") pod \"perf-node-gather-daemonset-zbfxs\" (UID: \"da82d61d-3311-4868-b781-ff2c489f3ffd\") " pod="openshift-must-gather-5v74r/perf-node-gather-daemonset-zbfxs" Sep 07 16:53:06.406591 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:06.406535 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/da82d61d-3311-4868-b781-ff2c489f3ffd-proc\") pod \"perf-node-gather-daemonset-zbfxs\" (UID: \"da82d61d-3311-4868-b781-ff2c489f3ffd\") " pod="openshift-must-gather-5v74r/perf-node-gather-daemonset-zbfxs" Sep 07 16:53:06.406766 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:06.406608 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/da82d61d-3311-4868-b781-ff2c489f3ffd-lib-modules\") pod \"perf-node-gather-daemonset-zbfxs\" (UID: \"da82d61d-3311-4868-b781-ff2c489f3ffd\") " pod="openshift-must-gather-5v74r/perf-node-gather-daemonset-zbfxs" Sep 07 16:53:06.406766 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:06.406613 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/da82d61d-3311-4868-b781-ff2c489f3ffd-podres\") pod \"perf-node-gather-daemonset-zbfxs\" (UID: \"da82d61d-3311-4868-b781-ff2c489f3ffd\") " pod="openshift-must-gather-5v74r/perf-node-gather-daemonset-zbfxs" Sep 07 16:53:06.414013 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:06.413984 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t6wx\" (UniqueName: \"kubernetes.io/projected/da82d61d-3311-4868-b781-ff2c489f3ffd-kube-api-access-6t6wx\") pod \"perf-node-gather-daemonset-zbfxs\" (UID: \"da82d61d-3311-4868-b781-ff2c489f3ffd\") " pod="openshift-must-gather-5v74r/perf-node-gather-daemonset-zbfxs" Sep 07 16:53:06.540200 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:06.540108 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5v74r/perf-node-gather-daemonset-zbfxs" Sep 07 16:53:06.807551 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:06.807460 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5v74r/perf-node-gather-daemonset-zbfxs"] Sep 07 16:53:06.810607 ip-10-0-135-242 kubenswrapper[2580]: W0907 16:53:06.810574 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podda82d61d_3311_4868_b781_ff2c489f3ffd.slice/crio-3baae73411259f849912051f98638d05d2153ca19db0ca0e621215059f4f80b1 WatchSource:0}: Error finding container 3baae73411259f849912051f98638d05d2153ca19db0ca0e621215059f4f80b1: Status 404 returned error can't find the container with id 3baae73411259f849912051f98638d05d2153ca19db0ca0e621215059f4f80b1 Sep 07 16:53:06.895301 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:06.895263 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5v74r/perf-node-gather-daemonset-zbfxs" event={"ID":"da82d61d-3311-4868-b781-ff2c489f3ffd","Type":"ContainerStarted","Data":"3baae73411259f849912051f98638d05d2153ca19db0ca0e621215059f4f80b1"} Sep 07 16:53:07.902079 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:07.902042 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5v74r/perf-node-gather-daemonset-zbfxs" event={"ID":"da82d61d-3311-4868-b781-ff2c489f3ffd","Type":"ContainerStarted","Data":"20acdac3c452c2e4b64e76c73feaab20798c611eab62e208bf007b5761a39278"} Sep 07 16:53:07.902501 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:07.902160 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5v74r/perf-node-gather-daemonset-zbfxs" Sep 07 16:53:07.915317 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:07.915231 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5v74r/perf-node-gather-daemonset-zbfxs" podStartSLOduration=1.915213501 podStartE2EDuration="1.915213501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 16:53:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 16:53:07.914941815 +0000 UTC m=+2016.950164877" watchObservedRunningTime="2026-09-07 16:53:07.915213501 +0000 UTC m=+2016.950436573" Sep 07 16:53:08.423799 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:08.423768 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-l2gc8_5bc83c7d-9a35-4c79-82a4-bdad68c35006/dns/0.log" Sep 07 16:53:08.442461 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:08.442421 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-l2gc8_5bc83c7d-9a35-4c79-82a4-bdad68c35006/kube-rbac-proxy/0.log" Sep 07 16:53:08.558003 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:08.557973 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kcb7q_553ed3c9-3691-4532-8a44-702ea20b17a7/dns-node-resolver/0.log" Sep 07 16:53:09.076896 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:09.076863 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-8d749d949-6k8qs_945c7b16-2a9a-450e-8068-2016fb348282/registry/0.log" Sep 07 16:53:09.099636 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:09.099605 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-g9wgk_8c534a46-aee8-4f33-a447-c049fb3d3f04/node-ca/0.log" Sep 07 16:53:10.029462 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:10.029430 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg_8fe982b2-5675-4a4b-b8a7-5cc0c397480c/istio-proxy/0.log" Sep 07 16:53:10.560896 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:10.560867 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-2rc48_182ffcc1-816e-4871-a357-6b9ac07e4481/discovery/0.log" Sep 07 16:53:10.692974 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:10.692939 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-wpflr_174d0f3e-5f11-4428-8839-784bff454949/payload-pre-processing/0.log" Sep 07 16:53:11.603945 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:11.603913 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vhwvg_88d5615b-94e6-42ad-bf64-a9e443f8a1ba/serve-healthcheck-canary/0.log" Sep 07 16:53:12.065388 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:12.065357 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kwckr_89ae6799-1f6e-422c-82b3-ab64bf3967b4/kube-rbac-proxy/0.log" Sep 07 16:53:12.083489 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:12.083448 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kwckr_89ae6799-1f6e-422c-82b3-ab64bf3967b4/exporter/0.log" Sep 07 16:53:12.102164 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:12.102131 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kwckr_89ae6799-1f6e-422c-82b3-ab64bf3967b4/extractor/0.log" Sep 07 16:53:13.917172 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:13.917144 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5v74r/perf-node-gather-daemonset-zbfxs" Sep 07 16:53:14.791434 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:14.791402 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7766cc5489-hcs8n_8c454a42-b87a-4061-b104-522b821e3978/manager/0.log" Sep 07 16:53:14.808502 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:14.808475 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7766cc5489-hcs8n_8c454a42-b87a-4061-b104-522b821e3978/copy-manifests/0.log" Sep 07 16:53:16.160909 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:16.160876 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-57d49c86b4-xp4kf_363da8d9-ef7a-4eee-9ef4-1f7f4fc60414/manager/0.log" Sep 07 16:53:16.182524 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:16.182494 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-ppn5t_8759d9b5-5657-4082-8318-6340fd432a25/openshift-lws-operator/0.log" Sep 07 16:53:16.538655 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:16.538621 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg_8fe982b2-5675-4a4b-b8a7-5cc0c397480c/istio-proxy/0.log" Sep 07 16:53:17.074053 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:17.073999 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-2rc48_182ffcc1-816e-4871-a357-6b9ac07e4481/discovery/0.log" Sep 07 16:53:17.168598 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:17.168576 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-wpflr_174d0f3e-5f11-4428-8839-784bff454949/payload-pre-processing/0.log" Sep 07 16:53:18.659221 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:18.659187 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-884cg_8fe982b2-5675-4a4b-b8a7-5cc0c397480c/istio-proxy/0.log" Sep 07 16:53:19.199552 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:19.199515 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-2rc48_182ffcc1-816e-4871-a357-6b9ac07e4481/discovery/0.log" Sep 07 16:53:19.305467 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:19.305431 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-wpflr_174d0f3e-5f11-4428-8839-784bff454949/payload-pre-processing/0.log" Sep 07 16:53:20.959167 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:20.959136 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-fxdvm_509635fb-d929-4bbd-bdfc-f875c2771bf6/migrator/0.log" Sep 07 16:53:20.976400 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:20.976367 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-fxdvm_509635fb-d929-4bbd-bdfc-f875c2771bf6/graceful-termination/0.log" Sep 07 16:53:22.467631 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:22.467573 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7tdcz_2e77c59f-ef45-4fb8-99be-5fdccb594e0c/kube-multus-additional-cni-plugins/0.log" Sep 07 16:53:22.487743 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:22.487717 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7tdcz_2e77c59f-ef45-4fb8-99be-5fdccb594e0c/egress-router-binary-copy/0.log" Sep 07 16:53:22.504824 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:22.504792 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7tdcz_2e77c59f-ef45-4fb8-99be-5fdccb594e0c/cni-plugins/0.log" Sep 07 16:53:22.522515 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:22.522487 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7tdcz_2e77c59f-ef45-4fb8-99be-5fdccb594e0c/bond-cni-plugin/0.log" Sep 07 16:53:22.539561 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:22.539536 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7tdcz_2e77c59f-ef45-4fb8-99be-5fdccb594e0c/routeoverride-cni/0.log" Sep 07 16:53:22.556580 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:22.556552 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7tdcz_2e77c59f-ef45-4fb8-99be-5fdccb594e0c/whereabouts-cni-bincopy/0.log" Sep 07 16:53:22.573919 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:22.573885 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7tdcz_2e77c59f-ef45-4fb8-99be-5fdccb594e0c/whereabouts-cni/0.log" Sep 07 16:53:22.748401 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:22.748319 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-t4bpj_140ff0c3-223c-4f7a-9142-fc8e999d6449/kube-multus/0.log" Sep 07 16:53:22.814245 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:22.814216 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-lhv86_599e026d-7338-4319-a1d7-7b714bbb6e58/network-metrics-daemon/0.log" Sep 07 16:53:22.827792 ip-10-0-135-242 kubenswrapper[2580]: I0907 16:53:22.827757 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-lhv86_599e026d-7338-4319-a1d7-7b714bbb6e58/kube-rbac-proxy/0.log"