Sep 11 18:13:09.145739 ip-10-0-138-128 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 11 18:13:09.145750 ip-10-0-138-128 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 11 18:13:09.145757 ip-10-0-138-128 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 11 18:13:09.145928 ip-10-0-138-128 systemd[1]: Failed to start Kubernetes Kubelet. Sep 11 18:13:19.299628 ip-10-0-138-128 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 11 18:13:19.299643 ip-10-0-138-128 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot ed5e202620ef48ef91b438e8f677ae51 -- Sep 11 18:15:16.863451 ip-10-0-138-128 systemd[1]: Starting Kubernetes Kubelet... Sep 11 18:15:17.259050 ip-10-0-138-128 kubenswrapper[2561]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 18:15:17.259050 ip-10-0-138-128 kubenswrapper[2561]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 11 18:15:17.259050 ip-10-0-138-128 kubenswrapper[2561]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 18:15:17.259050 ip-10-0-138-128 kubenswrapper[2561]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 11 18:15:17.259050 ip-10-0-138-128 kubenswrapper[2561]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 18:15:17.261942 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.261847 2561 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 11 18:15:17.265946 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.265930 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 18:15:17.265946 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.265947 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 18:15:17.266015 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.265951 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 18:15:17.266015 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.265954 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 18:15:17.266015 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.265958 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 18:15:17.266015 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.265962 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 18:15:17.266015 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.265965 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 18:15:17.266015 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.265968 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 18:15:17.266015 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.265971 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 18:15:17.266015 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.265974 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 18:15:17.266015 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.265977 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 18:15:17.266015 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.265979 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 18:15:17.266015 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.265982 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 18:15:17.266015 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.265984 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 18:15:17.266015 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.265988 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 18:15:17.266015 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.265993 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 18:15:17.266015 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.265996 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 18:15:17.266015 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.265998 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 18:15:17.266015 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266001 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 18:15:17.266015 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266003 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 18:15:17.266015 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266012 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 18:15:17.266539 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266015 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 18:15:17.266539 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266018 2561 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 18:15:17.266539 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266021 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 18:15:17.266539 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266024 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 18:15:17.266539 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266027 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 18:15:17.266539 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266030 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 18:15:17.266539 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266032 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 18:15:17.266539 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266035 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 18:15:17.266539 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266038 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 18:15:17.266539 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266040 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 18:15:17.266539 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266043 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 18:15:17.266539 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266048 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 18:15:17.266539 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266052 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 18:15:17.266539 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266055 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 18:15:17.266539 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266058 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 18:15:17.266539 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266061 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 18:15:17.266539 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266063 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 18:15:17.266539 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266066 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 18:15:17.266539 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266069 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 18:15:17.267332 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266071 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 18:15:17.267332 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266073 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 18:15:17.267332 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266076 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 18:15:17.267332 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266078 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 18:15:17.267332 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266081 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 18:15:17.267332 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266083 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 18:15:17.267332 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266085 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 18:15:17.267332 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266088 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 18:15:17.267332 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266090 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 18:15:17.267332 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266093 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 18:15:17.267332 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266096 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 18:15:17.267332 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266098 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 18:15:17.267332 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266100 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 18:15:17.267332 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266104 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 18:15:17.267332 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266108 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 18:15:17.267332 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266110 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 18:15:17.267332 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266113 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 18:15:17.267332 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266116 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 18:15:17.267332 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266119 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 18:15:17.267332 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266121 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 18:15:17.267823 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266125 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 18:15:17.267823 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266140 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 18:15:17.267823 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266143 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 18:15:17.267823 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266146 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 18:15:17.267823 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266149 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 18:15:17.267823 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266152 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 18:15:17.267823 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266155 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 18:15:17.267823 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266157 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 18:15:17.267823 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266160 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 18:15:17.267823 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266162 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 18:15:17.267823 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266165 2561 feature_gate.go:328] unrecognized feature gate: Example Sep 11 18:15:17.267823 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266168 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 18:15:17.267823 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266171 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 18:15:17.267823 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266173 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 18:15:17.267823 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266176 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 18:15:17.267823 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266180 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 18:15:17.267823 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266182 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 18:15:17.267823 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266185 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 18:15:17.267823 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266188 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 18:15:17.267823 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266191 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 18:15:17.268333 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266193 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 18:15:17.268333 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266196 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 18:15:17.268333 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266199 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 18:15:17.268333 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266202 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 18:15:17.268333 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266204 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 18:15:17.268333 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266207 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 18:15:17.268333 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266211 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 18:15:17.268333 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266214 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 18:15:17.268333 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266217 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 18:15:17.268333 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266219 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 18:15:17.268333 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266222 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 18:15:17.268333 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266225 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 18:15:17.268333 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266228 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 18:15:17.268333 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266230 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 18:15:17.268333 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266233 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 18:15:17.268333 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266235 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 18:15:17.268333 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266238 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 18:15:17.268333 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266240 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 18:15:17.268333 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.266243 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 18:15:17.268333 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268007 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 18:15:17.268816 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268019 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 18:15:17.268816 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268023 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 18:15:17.268816 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268026 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 18:15:17.268816 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268029 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 18:15:17.268816 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268034 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 18:15:17.268816 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268039 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 18:15:17.268816 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268042 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 18:15:17.268816 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268045 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 18:15:17.268816 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268048 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 18:15:17.268816 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268058 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 18:15:17.268816 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268061 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 18:15:17.268816 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268064 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 18:15:17.268816 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268067 2561 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 18:15:17.268816 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268070 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 18:15:17.268816 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268072 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 18:15:17.268816 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268075 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 18:15:17.268816 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268078 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 18:15:17.268816 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268081 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 18:15:17.268816 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268083 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 18:15:17.268816 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268086 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 18:15:17.269329 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268089 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 18:15:17.269329 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268091 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 18:15:17.269329 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268094 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 18:15:17.269329 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268096 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 18:15:17.269329 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268099 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 18:15:17.269329 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268102 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 18:15:17.269329 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268104 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 18:15:17.269329 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268107 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 18:15:17.269329 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268109 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 18:15:17.269329 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268112 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 18:15:17.269329 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268115 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 18:15:17.269329 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268118 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 18:15:17.269329 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268121 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 18:15:17.269329 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268124 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 18:15:17.269329 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268126 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 18:15:17.269329 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268146 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 18:15:17.269329 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268150 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 18:15:17.269329 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268153 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 18:15:17.269329 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268156 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 18:15:17.269329 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268159 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 18:15:17.269821 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268161 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 18:15:17.269821 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268164 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 18:15:17.269821 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268167 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 18:15:17.269821 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268169 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 18:15:17.269821 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268172 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 18:15:17.269821 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268175 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 18:15:17.269821 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268177 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 18:15:17.269821 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268180 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 18:15:17.269821 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268182 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 18:15:17.269821 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268185 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 18:15:17.269821 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268188 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 18:15:17.269821 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268190 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 18:15:17.269821 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268193 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 18:15:17.269821 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268195 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 18:15:17.269821 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268203 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 18:15:17.269821 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268206 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 18:15:17.269821 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268208 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 18:15:17.269821 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268211 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 18:15:17.269821 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268213 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 18:15:17.270304 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268216 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 18:15:17.270304 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268218 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 18:15:17.270304 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268221 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 18:15:17.270304 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268223 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 18:15:17.270304 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268226 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 18:15:17.270304 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268229 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 18:15:17.270304 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268232 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 18:15:17.270304 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268234 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 18:15:17.270304 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268237 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 18:15:17.270304 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268240 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 18:15:17.270304 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268242 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 18:15:17.270304 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268245 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 18:15:17.270304 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268247 2561 feature_gate.go:328] unrecognized feature gate: Example Sep 11 18:15:17.270304 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268250 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 18:15:17.270304 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268253 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 18:15:17.270304 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268255 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 18:15:17.270304 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268259 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 18:15:17.270304 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268263 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 18:15:17.270304 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268265 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 18:15:17.270304 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268268 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 18:15:17.270789 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268272 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 18:15:17.270789 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268275 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 18:15:17.270789 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268278 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 18:15:17.270789 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268282 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 18:15:17.270789 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268284 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 18:15:17.270789 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268287 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 18:15:17.270789 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268290 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 18:15:17.270789 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268293 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 18:15:17.270789 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268296 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 18:15:17.270789 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268298 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 18:15:17.270789 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268301 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 18:15:17.270789 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268303 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 18:15:17.270789 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268306 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 18:15:17.270789 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268309 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 18:15:17.270789 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268311 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 18:15:17.270789 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268314 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 18:15:17.270789 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268316 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 18:15:17.270789 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268319 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 18:15:17.270789 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268322 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 18:15:17.270789 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268702 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 18:15:17.271393 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268707 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 18:15:17.271393 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268710 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 18:15:17.271393 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268713 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 18:15:17.271393 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268716 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 18:15:17.271393 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268719 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 18:15:17.271393 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268722 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 18:15:17.271393 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268724 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 18:15:17.271393 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268727 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 18:15:17.271393 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268729 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 18:15:17.271393 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268734 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 18:15:17.271393 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268737 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 18:15:17.271393 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268741 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 18:15:17.271393 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268744 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 18:15:17.271393 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268747 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 18:15:17.271393 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268749 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 18:15:17.271393 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268753 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 18:15:17.271393 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268756 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 18:15:17.271393 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268759 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 18:15:17.271393 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268762 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 18:15:17.271846 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268765 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 18:15:17.271846 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268767 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 18:15:17.271846 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268771 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 18:15:17.271846 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268775 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 18:15:17.271846 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268777 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 18:15:17.271846 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268780 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 18:15:17.271846 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268783 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 18:15:17.271846 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268786 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 18:15:17.271846 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268789 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 18:15:17.271846 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268792 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 18:15:17.271846 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268795 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 18:15:17.271846 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268798 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 18:15:17.271846 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268801 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 18:15:17.271846 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268804 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 18:15:17.271846 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268806 2561 feature_gate.go:328] unrecognized feature gate: Example Sep 11 18:15:17.271846 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268809 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 18:15:17.271846 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268811 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 18:15:17.271846 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268814 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 18:15:17.271846 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268816 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 18:15:17.271846 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268819 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 18:15:17.272346 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268822 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 18:15:17.272346 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268824 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 18:15:17.272346 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268827 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 18:15:17.272346 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268830 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 18:15:17.272346 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268832 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 18:15:17.272346 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268834 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 18:15:17.272346 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268837 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 18:15:17.272346 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268839 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 18:15:17.272346 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268842 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 18:15:17.272346 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268845 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 18:15:17.272346 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268848 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 18:15:17.272346 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268850 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 18:15:17.272346 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268853 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 18:15:17.272346 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268855 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 18:15:17.272346 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268858 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 18:15:17.272346 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268860 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 18:15:17.272346 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268863 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 18:15:17.272346 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268865 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 18:15:17.272346 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268868 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 18:15:17.272346 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268871 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 18:15:17.272848 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268873 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 18:15:17.272848 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268876 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 18:15:17.272848 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268879 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 18:15:17.272848 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268883 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 18:15:17.272848 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268885 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 18:15:17.272848 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268888 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 18:15:17.272848 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268891 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 18:15:17.272848 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268893 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 18:15:17.272848 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268896 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 18:15:17.272848 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268899 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 18:15:17.272848 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268901 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 18:15:17.272848 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268904 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 18:15:17.272848 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268906 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 18:15:17.272848 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268909 2561 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 18:15:17.272848 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268911 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 18:15:17.272848 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268913 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 18:15:17.272848 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268917 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 18:15:17.272848 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268920 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 18:15:17.272848 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268924 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 18:15:17.272848 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268926 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 18:15:17.273361 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268929 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 18:15:17.273361 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268932 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 18:15:17.273361 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268935 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 18:15:17.273361 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268937 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 18:15:17.273361 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268940 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 18:15:17.273361 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268942 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 18:15:17.273361 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268945 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 18:15:17.273361 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268947 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 18:15:17.273361 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268950 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 18:15:17.273361 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268952 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 18:15:17.273361 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268955 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 18:15:17.273361 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268958 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 18:15:17.273361 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268960 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 18:15:17.273361 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268963 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 18:15:17.273361 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268966 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 18:15:17.273361 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268969 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 18:15:17.273361 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268971 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 18:15:17.273361 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268974 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 18:15:17.273361 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.268976 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 18:15:17.273361 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269069 2561 flags.go:64] FLAG: --address="0.0.0.0" Sep 11 18:15:17.273881 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269119 2561 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 11 18:15:17.273881 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269128 2561 flags.go:64] FLAG: --anonymous-auth="true" Sep 11 18:15:17.273881 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269146 2561 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 11 18:15:17.273881 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269151 2561 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 11 18:15:17.273881 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269154 2561 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 11 18:15:17.273881 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269158 2561 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 11 18:15:17.273881 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269163 2561 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 11 18:15:17.273881 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269166 2561 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 11 18:15:17.273881 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269169 2561 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 11 18:15:17.273881 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269173 2561 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 11 18:15:17.273881 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269177 2561 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 11 18:15:17.273881 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269180 2561 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 11 18:15:17.273881 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269183 2561 flags.go:64] FLAG: --cgroup-root="" Sep 11 18:15:17.273881 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269185 2561 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 11 18:15:17.273881 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269188 2561 flags.go:64] FLAG: --client-ca-file="" Sep 11 18:15:17.273881 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269192 2561 flags.go:64] FLAG: --cloud-provider="external" Sep 11 18:15:17.273881 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269195 2561 flags.go:64] FLAG: --cluster-dns="[]" Sep 11 18:15:17.273881 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269199 2561 flags.go:64] FLAG: --cluster-domain="" Sep 11 18:15:17.273881 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269203 2561 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 11 18:15:17.273881 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269206 2561 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 11 18:15:17.273881 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269209 2561 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 11 18:15:17.273881 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269212 2561 flags.go:64] FLAG: --container-log-max-files="5" Sep 11 18:15:17.273881 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269222 2561 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 11 18:15:17.273881 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269225 2561 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 11 18:15:17.274493 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269227 2561 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 11 18:15:17.274493 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269231 2561 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 11 18:15:17.274493 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269235 2561 flags.go:64] FLAG: --contention-profiling="false" Sep 11 18:15:17.274493 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269238 2561 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 11 18:15:17.274493 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269241 2561 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 11 18:15:17.274493 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269244 2561 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 11 18:15:17.274493 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269248 2561 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 11 18:15:17.274493 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269252 2561 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 11 18:15:17.274493 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269255 2561 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 11 18:15:17.274493 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269258 2561 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 11 18:15:17.274493 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269261 2561 flags.go:64] FLAG: --enable-load-reader="false" Sep 11 18:15:17.274493 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269264 2561 flags.go:64] FLAG: --enable-server="true" Sep 11 18:15:17.274493 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269267 2561 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 11 18:15:17.274493 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269272 2561 flags.go:64] FLAG: --event-burst="100" Sep 11 18:15:17.274493 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269275 2561 flags.go:64] FLAG: --event-qps="50" Sep 11 18:15:17.274493 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269278 2561 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 11 18:15:17.274493 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269281 2561 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 11 18:15:17.274493 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269284 2561 flags.go:64] FLAG: --eviction-hard="" Sep 11 18:15:17.274493 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269288 2561 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 11 18:15:17.274493 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269291 2561 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 11 18:15:17.274493 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269294 2561 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 11 18:15:17.274493 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269297 2561 flags.go:64] FLAG: --eviction-soft="" Sep 11 18:15:17.274493 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269300 2561 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 11 18:15:17.274493 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269303 2561 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 11 18:15:17.274493 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269307 2561 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 11 18:15:17.275117 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269310 2561 flags.go:64] FLAG: --experimental-mounter-path="" Sep 11 18:15:17.275117 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269313 2561 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 11 18:15:17.275117 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269316 2561 flags.go:64] FLAG: --fail-swap-on="true" Sep 11 18:15:17.275117 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269319 2561 flags.go:64] FLAG: --feature-gates="" Sep 11 18:15:17.275117 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269322 2561 flags.go:64] FLAG: --file-check-frequency="20s" Sep 11 18:15:17.275117 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269325 2561 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 11 18:15:17.275117 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269328 2561 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 11 18:15:17.275117 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269332 2561 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 11 18:15:17.275117 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269336 2561 flags.go:64] FLAG: --healthz-port="10248" Sep 11 18:15:17.275117 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269340 2561 flags.go:64] FLAG: --help="false" Sep 11 18:15:17.275117 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269343 2561 flags.go:64] FLAG: --hostname-override="ip-10-0-138-128.ec2.internal" Sep 11 18:15:17.275117 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269346 2561 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 11 18:15:17.275117 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269349 2561 flags.go:64] FLAG: --http-check-frequency="20s" Sep 11 18:15:17.275117 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269352 2561 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 11 18:15:17.275117 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269356 2561 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 11 18:15:17.275117 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269359 2561 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 11 18:15:17.275117 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269362 2561 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 11 18:15:17.275117 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269365 2561 flags.go:64] FLAG: --image-service-endpoint="" Sep 11 18:15:17.275117 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269368 2561 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 11 18:15:17.275117 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269371 2561 flags.go:64] FLAG: --kube-api-burst="100" Sep 11 18:15:17.275117 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269374 2561 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 11 18:15:17.275117 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269377 2561 flags.go:64] FLAG: --kube-api-qps="50" Sep 11 18:15:17.275117 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269379 2561 flags.go:64] FLAG: --kube-reserved="" Sep 11 18:15:17.275117 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269382 2561 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 11 18:15:17.275712 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269385 2561 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 11 18:15:17.275712 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269388 2561 flags.go:64] FLAG: --kubelet-cgroups="" Sep 11 18:15:17.275712 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269391 2561 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 11 18:15:17.275712 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269393 2561 flags.go:64] FLAG: --lock-file="" Sep 11 18:15:17.275712 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269396 2561 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 11 18:15:17.275712 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269399 2561 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 11 18:15:17.275712 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269402 2561 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 11 18:15:17.275712 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269407 2561 flags.go:64] FLAG: --log-json-split-stream="false" Sep 11 18:15:17.275712 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269415 2561 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 11 18:15:17.275712 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269417 2561 flags.go:64] FLAG: --log-text-split-stream="false" Sep 11 18:15:17.275712 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269420 2561 flags.go:64] FLAG: --logging-format="text" Sep 11 18:15:17.275712 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269423 2561 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 11 18:15:17.275712 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269427 2561 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 11 18:15:17.275712 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269430 2561 flags.go:64] FLAG: --manifest-url="" Sep 11 18:15:17.275712 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269432 2561 flags.go:64] FLAG: --manifest-url-header="" Sep 11 18:15:17.275712 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269437 2561 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 11 18:15:17.275712 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269442 2561 flags.go:64] FLAG: --max-open-files="1000000" Sep 11 18:15:17.275712 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269448 2561 flags.go:64] FLAG: --max-pods="110" Sep 11 18:15:17.275712 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269451 2561 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 11 18:15:17.275712 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269454 2561 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 11 18:15:17.275712 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269457 2561 flags.go:64] FLAG: --memory-manager-policy="None" Sep 11 18:15:17.275712 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269460 2561 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 11 18:15:17.275712 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269463 2561 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 11 18:15:17.275712 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269466 2561 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 11 18:15:17.275712 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269469 2561 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 11 18:15:17.276338 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269477 2561 flags.go:64] FLAG: --node-status-max-images="50" Sep 11 18:15:17.276338 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269480 2561 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 11 18:15:17.276338 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269483 2561 flags.go:64] FLAG: --oom-score-adj="-999" Sep 11 18:15:17.276338 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269486 2561 flags.go:64] FLAG: --pod-cidr="" Sep 11 18:15:17.276338 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269489 2561 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 11 18:15:17.276338 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269495 2561 flags.go:64] FLAG: --pod-manifest-path="" Sep 11 18:15:17.276338 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269498 2561 flags.go:64] FLAG: --pod-max-pids="-1" Sep 11 18:15:17.276338 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269501 2561 flags.go:64] FLAG: --pods-per-core="0" Sep 11 18:15:17.276338 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269504 2561 flags.go:64] FLAG: --port="10250" Sep 11 18:15:17.276338 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269507 2561 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 11 18:15:17.276338 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269510 2561 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b232049eeba6e362" Sep 11 18:15:17.276338 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269523 2561 flags.go:64] FLAG: --qos-reserved="" Sep 11 18:15:17.276338 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269526 2561 flags.go:64] FLAG: --read-only-port="10255" Sep 11 18:15:17.276338 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269529 2561 flags.go:64] FLAG: --register-node="true" Sep 11 18:15:17.276338 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269532 2561 flags.go:64] FLAG: --register-with-taints="" Sep 11 18:15:17.276338 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269536 2561 flags.go:64] FLAG: --registry-burst="10" Sep 11 18:15:17.276338 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269540 2561 flags.go:64] FLAG: --registry-qps="5" Sep 11 18:15:17.276338 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269543 2561 flags.go:64] FLAG: --reserved-cpus="" Sep 11 18:15:17.276338 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269546 2561 flags.go:64] FLAG: --reserved-memory="" Sep 11 18:15:17.276338 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269549 2561 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 11 18:15:17.276338 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269552 2561 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 11 18:15:17.276338 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269555 2561 flags.go:64] FLAG: --rotate-certificates="false" Sep 11 18:15:17.276338 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269559 2561 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 11 18:15:17.276338 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269563 2561 flags.go:64] FLAG: --runonce="false" Sep 11 18:15:17.276338 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269566 2561 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 11 18:15:17.276983 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269572 2561 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 11 18:15:17.276983 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269575 2561 flags.go:64] FLAG: --seccomp-default="false" Sep 11 18:15:17.276983 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269578 2561 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 11 18:15:17.276983 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269581 2561 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 11 18:15:17.276983 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269585 2561 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 11 18:15:17.276983 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269588 2561 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 11 18:15:17.276983 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269591 2561 flags.go:64] FLAG: --storage-driver-password="root" Sep 11 18:15:17.276983 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269594 2561 flags.go:64] FLAG: --storage-driver-secure="false" Sep 11 18:15:17.276983 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269597 2561 flags.go:64] FLAG: --storage-driver-table="stats" Sep 11 18:15:17.276983 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269600 2561 flags.go:64] FLAG: --storage-driver-user="root" Sep 11 18:15:17.276983 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269603 2561 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 11 18:15:17.276983 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269606 2561 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 11 18:15:17.276983 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269609 2561 flags.go:64] FLAG: --system-cgroups="" Sep 11 18:15:17.276983 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269612 2561 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 11 18:15:17.276983 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269617 2561 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 11 18:15:17.276983 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269620 2561 flags.go:64] FLAG: --tls-cert-file="" Sep 11 18:15:17.276983 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269622 2561 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 11 18:15:17.276983 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269627 2561 flags.go:64] FLAG: --tls-min-version="" Sep 11 18:15:17.276983 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269629 2561 flags.go:64] FLAG: --tls-private-key-file="" Sep 11 18:15:17.276983 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269632 2561 flags.go:64] FLAG: --topology-manager-policy="none" Sep 11 18:15:17.276983 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269635 2561 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 11 18:15:17.276983 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269638 2561 flags.go:64] FLAG: --topology-manager-scope="container" Sep 11 18:15:17.276983 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269641 2561 flags.go:64] FLAG: --v="2" Sep 11 18:15:17.276983 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269645 2561 flags.go:64] FLAG: --version="false" Sep 11 18:15:17.276983 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269649 2561 flags.go:64] FLAG: --vmodule="" Sep 11 18:15:17.277725 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269653 2561 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 11 18:15:17.277725 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.269656 2561 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 11 18:15:17.277725 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269779 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 18:15:17.277725 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269782 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 18:15:17.277725 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269785 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 18:15:17.277725 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269790 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 18:15:17.277725 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269793 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 18:15:17.277725 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269798 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 18:15:17.277725 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269801 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 18:15:17.277725 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269803 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 18:15:17.277725 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269808 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 18:15:17.277725 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269811 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 18:15:17.277725 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269814 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 18:15:17.277725 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269816 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 18:15:17.277725 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269819 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 18:15:17.277725 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269821 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 18:15:17.277725 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269824 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 18:15:17.277725 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269827 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 18:15:17.277725 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269830 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 18:15:17.277725 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269833 2561 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 18:15:17.278227 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269836 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 18:15:17.278227 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269838 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 18:15:17.278227 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269841 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 18:15:17.278227 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269843 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 18:15:17.278227 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269846 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 18:15:17.278227 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269850 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 18:15:17.278227 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269853 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 18:15:17.278227 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269856 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 18:15:17.278227 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269859 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 18:15:17.278227 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269862 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 18:15:17.278227 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269865 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 18:15:17.278227 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269868 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 18:15:17.278227 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269871 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 18:15:17.278227 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269873 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 18:15:17.278227 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269876 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 18:15:17.278227 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269879 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 18:15:17.278227 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269881 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 18:15:17.278227 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269885 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 18:15:17.278227 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269888 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 18:15:17.278695 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269892 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 18:15:17.278695 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269894 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 18:15:17.278695 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269897 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 18:15:17.278695 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269899 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 18:15:17.278695 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269902 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 18:15:17.278695 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269904 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 18:15:17.278695 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269907 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 18:15:17.278695 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269910 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 18:15:17.278695 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269913 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 18:15:17.278695 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269915 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 18:15:17.278695 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269918 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 18:15:17.278695 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269921 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 18:15:17.278695 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269923 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 18:15:17.278695 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269925 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 18:15:17.278695 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269928 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 18:15:17.278695 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269931 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 18:15:17.278695 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269934 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 18:15:17.278695 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269936 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 18:15:17.278695 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269939 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 18:15:17.279177 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269941 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 18:15:17.279177 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269944 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 18:15:17.279177 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269947 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 18:15:17.279177 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269949 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 18:15:17.279177 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269952 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 18:15:17.279177 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269954 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 18:15:17.279177 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269957 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 18:15:17.279177 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269960 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 18:15:17.279177 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269964 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 18:15:17.279177 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269968 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 18:15:17.279177 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269971 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 18:15:17.279177 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269976 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 18:15:17.279177 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269979 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 18:15:17.279177 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269983 2561 feature_gate.go:328] unrecognized feature gate: Example Sep 11 18:15:17.279177 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269986 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 18:15:17.279177 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269988 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 18:15:17.279177 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269991 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 18:15:17.279177 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269993 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 18:15:17.279177 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269996 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 18:15:17.279177 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.269999 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 18:15:17.279655 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.270002 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 18:15:17.279655 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.270004 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 18:15:17.279655 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.270007 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 18:15:17.279655 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.270010 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 18:15:17.279655 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.270017 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 18:15:17.279655 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.270020 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 18:15:17.279655 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.270023 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 18:15:17.279655 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.270025 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 18:15:17.279655 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.270028 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 18:15:17.279655 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.270031 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 18:15:17.279655 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.270033 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 18:15:17.279655 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.270036 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 18:15:17.279655 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.270038 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 18:15:17.279655 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.270041 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 18:15:17.279655 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.270044 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 18:15:17.279655 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.270047 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 18:15:17.279655 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.270049 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 18:15:17.279655 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.270052 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 18:15:17.279655 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.270054 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 18:15:17.279655 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.270057 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 18:15:17.280149 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.270060 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 18:15:17.280149 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.270062 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 18:15:17.280149 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.270065 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 18:15:17.280149 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.270970 2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 18:15:17.280149 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.278425 2561 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 11 18:15:17.280149 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.278442 2561 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 11 18:15:17.280149 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.279282 2561 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 11 18:15:17.280149 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.279290 2561 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 11 18:15:17.280149 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279348 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 18:15:17.280149 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279353 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 18:15:17.280149 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279356 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 18:15:17.280149 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279359 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 18:15:17.280149 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279362 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 18:15:17.280149 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279365 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 18:15:17.280507 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279368 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 18:15:17.280507 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279371 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 18:15:17.280507 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279374 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 18:15:17.280507 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279377 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 18:15:17.280507 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279380 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 18:15:17.280507 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279383 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 18:15:17.280507 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279387 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 18:15:17.280507 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279391 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 18:15:17.280507 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279394 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 18:15:17.280507 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279397 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 18:15:17.280507 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279399 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 18:15:17.280507 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279402 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 18:15:17.280507 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279405 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 18:15:17.280507 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279407 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 18:15:17.280507 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279410 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 18:15:17.280507 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279412 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 18:15:17.280507 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279415 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 18:15:17.280507 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279418 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 18:15:17.280507 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279420 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 18:15:17.280507 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279423 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 18:15:17.281014 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279425 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 18:15:17.281014 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279428 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 18:15:17.281014 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279431 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 18:15:17.281014 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279435 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 18:15:17.281014 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279438 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 18:15:17.281014 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279441 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 18:15:17.281014 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279444 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 18:15:17.281014 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279446 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 18:15:17.281014 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279449 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 18:15:17.281014 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279452 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 18:15:17.281014 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279454 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 18:15:17.281014 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279457 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 18:15:17.281014 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279460 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 18:15:17.281014 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279463 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 18:15:17.281014 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279465 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 18:15:17.281014 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279468 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 18:15:17.281014 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279472 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 18:15:17.281014 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279475 2561 feature_gate.go:328] unrecognized feature gate: Example Sep 11 18:15:17.281014 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279478 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 18:15:17.281544 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279481 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 18:15:17.281544 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279485 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 18:15:17.281544 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279488 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 18:15:17.281544 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279492 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 18:15:17.281544 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279495 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 18:15:17.281544 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279498 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 18:15:17.281544 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279501 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 18:15:17.281544 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279503 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 18:15:17.281544 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279506 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 18:15:17.281544 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279509 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 18:15:17.281544 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279512 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 18:15:17.281544 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279514 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 18:15:17.281544 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279517 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 18:15:17.281544 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279520 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 18:15:17.281544 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279523 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 18:15:17.281544 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279525 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 18:15:17.281544 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279529 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 18:15:17.281544 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279532 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 18:15:17.281544 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279535 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 18:15:17.282004 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279538 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 18:15:17.282004 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279540 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 18:15:17.282004 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279543 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 18:15:17.282004 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279546 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 18:15:17.282004 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279548 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 18:15:17.282004 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279551 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 18:15:17.282004 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279554 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 18:15:17.282004 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279557 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 18:15:17.282004 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279559 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 18:15:17.282004 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279562 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 18:15:17.282004 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279564 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 18:15:17.282004 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279567 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 18:15:17.282004 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279570 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 18:15:17.282004 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279572 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 18:15:17.282004 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279575 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 18:15:17.282004 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279578 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 18:15:17.282004 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279580 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 18:15:17.282004 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279583 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 18:15:17.282004 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279585 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 18:15:17.282004 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279588 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 18:15:17.282492 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279591 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 18:15:17.282492 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279593 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 18:15:17.282492 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279596 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 18:15:17.282492 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279598 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 18:15:17.282492 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279601 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 18:15:17.282492 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279603 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 18:15:17.282492 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279606 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 18:15:17.282492 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279608 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 18:15:17.282492 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279611 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 18:15:17.282492 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279614 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 18:15:17.282492 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279617 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 18:15:17.282492 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279620 2561 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 18:15:17.282492 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279623 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 18:15:17.282492 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279626 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 18:15:17.282492 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279629 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 18:15:17.282916 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.279634 2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 18:15:17.282916 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279776 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 18:15:17.282916 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279782 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 18:15:17.282916 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279785 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 18:15:17.282916 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279788 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 18:15:17.282916 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279791 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 18:15:17.282916 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279794 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 18:15:17.282916 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279797 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 18:15:17.282916 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279800 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 18:15:17.282916 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279802 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 18:15:17.282916 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279805 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 18:15:17.282916 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279808 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 18:15:17.282916 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279810 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 18:15:17.282916 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279813 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 18:15:17.282916 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279816 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 18:15:17.283286 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279818 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 18:15:17.283286 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279821 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 18:15:17.283286 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279823 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 18:15:17.283286 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279826 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 18:15:17.283286 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279829 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 18:15:17.283286 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279831 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 18:15:17.283286 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279834 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 18:15:17.283286 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279836 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 18:15:17.283286 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279839 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 18:15:17.283286 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279842 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 18:15:17.283286 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279844 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 18:15:17.283286 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279847 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 18:15:17.283286 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279850 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 18:15:17.283286 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279853 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 18:15:17.283286 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279856 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 18:15:17.283286 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279859 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 18:15:17.283286 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279861 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 18:15:17.283286 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279864 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 18:15:17.283286 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279867 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 18:15:17.283740 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279870 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 18:15:17.283740 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279872 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 18:15:17.283740 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279875 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 18:15:17.283740 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279877 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 18:15:17.283740 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279880 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 18:15:17.283740 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279882 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 18:15:17.283740 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279885 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 18:15:17.283740 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279889 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 18:15:17.283740 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279892 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 18:15:17.283740 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279895 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 18:15:17.283740 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279898 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 18:15:17.283740 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279901 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 18:15:17.283740 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279904 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 18:15:17.283740 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279908 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 18:15:17.283740 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279911 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 18:15:17.283740 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279915 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 18:15:17.283740 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279918 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 18:15:17.283740 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279921 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 18:15:17.283740 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279924 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 18:15:17.284223 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279928 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 18:15:17.284223 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279931 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 18:15:17.284223 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279933 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 18:15:17.284223 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279936 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 18:15:17.284223 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279939 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 18:15:17.284223 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279941 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 18:15:17.284223 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279944 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 18:15:17.284223 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279947 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 18:15:17.284223 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279949 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 18:15:17.284223 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279952 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 18:15:17.284223 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279955 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 18:15:17.284223 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279957 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 18:15:17.284223 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279960 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 18:15:17.284223 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279962 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 18:15:17.284223 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279965 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 18:15:17.284223 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279967 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 18:15:17.284223 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279970 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 18:15:17.284223 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279972 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 18:15:17.284223 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279975 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 18:15:17.284223 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279977 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 18:15:17.284702 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279980 2561 feature_gate.go:328] unrecognized feature gate: Example Sep 11 18:15:17.284702 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279982 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 18:15:17.284702 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279985 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 18:15:17.284702 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279987 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 18:15:17.284702 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279990 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 18:15:17.284702 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279992 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 18:15:17.284702 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279995 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 18:15:17.284702 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.279998 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 18:15:17.284702 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.280001 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 18:15:17.284702 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.280003 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 18:15:17.284702 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.280006 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 18:15:17.284702 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.280008 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 18:15:17.284702 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.280011 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 18:15:17.284702 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.280014 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 18:15:17.284702 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.280016 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 18:15:17.284702 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.280019 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 18:15:17.284702 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.280021 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 18:15:17.284702 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.280024 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 18:15:17.284702 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.280027 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 18:15:17.284702 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.280029 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 18:15:17.285184 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.280032 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 18:15:17.285184 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.280034 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 18:15:17.285184 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.280038 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 18:15:17.285184 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.280040 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 18:15:17.285184 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.280043 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 18:15:17.285184 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.280045 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 18:15:17.285184 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:17.280048 2561 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 18:15:17.285184 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.280052 2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 18:15:17.285184 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.280173 2561 server.go:962] "Client rotation is on, will bootstrap in background" Sep 11 18:15:17.285184 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.282518 2561 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 11 18:15:17.285184 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.283614 2561 server.go:1019] "Starting client certificate rotation" Sep 11 18:15:17.285184 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.283711 2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 11 18:15:17.285184 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.283749 2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 11 18:15:17.286630 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.286616 2561 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 11 18:15:17.287779 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.287761 2561 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 11 18:15:17.298337 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.298317 2561 log.go:25] "Validated CRI v1 runtime API" Sep 11 18:15:17.301592 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.301576 2561 log.go:25] "Validated CRI v1 image API" Sep 11 18:15:17.302438 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.302416 2561 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 11 18:15:17.308409 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.308387 2561 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 b2ea8b09-2014-4286-a612-50feeda27f97:/dev/nvme0n1p4 b876f846-3b9d-44d0-859e-03b5b53408a5:/dev/nvme0n1p3] Sep 11 18:15:17.308495 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.308407 2561 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 11 18:15:17.314750 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.314630 2561 manager.go:216] Machine: {Timestamp:2026-09-11 18:15:17.313106923 +0000 UTC m=+0.349045715 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098640 MemoryCapacity:33164468224 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2753f256c8f36ae99229ed86c02e26 SystemUUID:ec2753f2-56c8-f36a-e992-29ed86c02e26 BootID:ed5e2026-20ef-48ef-91b4-38e8f677ae51 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582234112 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:aa:3f:06:b9:1f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:aa:3f:06:b9:1f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3a:6a:1f:fc:d1:e9 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164468224 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 11 18:15:17.314750 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.314740 2561 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 11 18:15:17.314866 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.314828 2561 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 11 18:15:17.316942 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.316926 2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 11 18:15:17.318276 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.318244 2561 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 11 18:15:17.318420 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.318279 2561 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-128.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 11 18:15:17.319092 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.319080 2561 topology_manager.go:138] "Creating topology manager with none policy" Sep 11 18:15:17.319092 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.319094 2561 container_manager_linux.go:309] "Creating device plugin manager" Sep 11 18:15:17.319191 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.319102 2561 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 11 18:15:17.319754 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.319744 2561 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 11 18:15:17.320903 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.320893 2561 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 11 18:15:17.325901 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.325889 2561 state_mem.go:36] "Initialized new in-memory state store" Sep 11 18:15:17.326028 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.326018 2561 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 11 18:15:17.326066 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.326051 2561 kubelet.go:486] "Attempting to sync node with API server" Sep 11 18:15:17.326095 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.326069 2561 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 11 18:15:17.326095 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.326085 2561 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 11 18:15:17.326095 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.326095 2561 kubelet.go:398] "Adding apiserver pod source" Sep 11 18:15:17.326239 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.326104 2561 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 11 18:15:17.326519 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.326506 2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 11 18:15:17.326555 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.326532 2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 11 18:15:17.330697 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.330678 2561 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 11 18:15:17.333091 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.333076 2561 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 11 18:15:17.333238 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.333106 2561 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 11 18:15:17.335247 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.335229 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 11 18:15:17.335318 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.335260 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 11 18:15:17.335318 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.335272 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 11 18:15:17.335318 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.335282 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 11 18:15:17.335318 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.335295 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 11 18:15:17.335318 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.335300 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 11 18:15:17.335318 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.335315 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 11 18:15:17.335484 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.335334 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 11 18:15:17.335484 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.335341 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 11 18:15:17.335484 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.335347 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 11 18:15:17.335484 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.335362 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 11 18:15:17.335484 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.335368 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 11 18:15:17.336232 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.336218 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 11 18:15:17.336292 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.336235 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 11 18:15:17.337397 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:17.337368 2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-128.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 11 18:15:17.337487 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:17.337413 2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 11 18:15:17.339386 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.339371 2561 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-128.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 11 18:15:17.340468 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.340450 2561 server.go:1268] "Started kubelet" Sep 11 18:15:17.340575 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.340545 2561 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 11 18:15:17.340743 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.340699 2561 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 11 18:15:17.340806 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.340763 2561 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 11 18:15:17.341482 ip-10-0-138-128 systemd[1]: Started Kubernetes Kubelet. Sep 11 18:15:17.343551 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.343526 2561 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 11 18:15:17.344899 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.344880 2561 server.go:310] "Adding debug handlers to kubelet server" Sep 11 18:15:17.345697 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:17.345668 2561 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 11 18:15:17.351066 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:17.349990 2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-128.ec2.internal.18d456ea8bc84a98 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-128.ec2.internal,UID:ip-10-0-138-128.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-128.ec2.internal,},FirstTimestamp:2026-09-11 18:15:17.340424856 +0000 UTC m=+0.376363648,LastTimestamp:2026-09-11 18:15:17.340424856 +0000 UTC m=+0.376363648,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-128.ec2.internal,}" Sep 11 18:15:17.351177 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.351162 2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 11 18:15:17.351846 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.351830 2561 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 11 18:15:17.351974 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:17.351958 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-128.ec2.internal\" not found" Sep 11 18:15:17.352630 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.352609 2561 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 11 18:15:17.352630 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.352612 2561 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 11 18:15:17.352743 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.352639 2561 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 11 18:15:17.352743 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.352734 2561 reconstruct.go:97] "Volume reconstruction finished" Sep 11 18:15:17.352743 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.352744 2561 reconciler.go:29] "Reconciler: start to sync state" Sep 11 18:15:17.352902 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.352886 2561 factory.go:153] Registering CRI-O factory Sep 11 18:15:17.352960 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.352908 2561 factory.go:223] Registration of the crio container factory successfully Sep 11 18:15:17.353006 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.352965 2561 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 11 18:15:17.353006 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.352975 2561 factory.go:55] Registering systemd factory Sep 11 18:15:17.353006 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.352984 2561 factory.go:223] Registration of the systemd container factory successfully Sep 11 18:15:17.353006 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.353000 2561 factory.go:103] Registering Raw factory Sep 11 18:15:17.353195 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.353014 2561 manager.go:1196] Started watching for new ooms in manager Sep 11 18:15:17.353434 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.353420 2561 manager.go:349] Starting recovery of all containers Sep 11 18:15:17.354822 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:17.354795 2561 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-128.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 11 18:15:17.354914 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:17.354847 2561 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 11 18:15:17.365497 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.365477 2561 manager.go:354] Recovery completed Sep 11 18:15:17.368578 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.368556 2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jjfxh" Sep 11 18:15:17.370472 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.370459 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 18:15:17.372664 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.372502 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-128.ec2.internal" event="NodeHasSufficientMemory" Sep 11 18:15:17.372728 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.372679 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-128.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 18:15:17.372728 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.372697 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-128.ec2.internal" event="NodeHasSufficientPID" Sep 11 18:15:17.373494 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.373477 2561 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 11 18:15:17.373494 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.373495 2561 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 11 18:15:17.373633 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.373523 2561 state_mem.go:36] "Initialized new in-memory state store" Sep 11 18:15:17.373633 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.373543 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 18:15:17.374528 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.374509 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-128.ec2.internal" event="NodeHasSufficientMemory" Sep 11 18:15:17.374615 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.374541 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-128.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 18:15:17.374615 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.374555 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-128.ec2.internal" event="NodeHasSufficientPID" Sep 11 18:15:17.374683 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:17.374583 2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-128.ec2.internal.18d456ea8db43d0d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-128.ec2.internal,UID:ip-10-0-138-128.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-128.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-128.ec2.internal,},FirstTimestamp:2026-09-11 18:15:17.372665101 +0000 UTC m=+0.408603899,LastTimestamp:2026-09-11 18:15:17.372665101 +0000 UTC m=+0.408603899,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-128.ec2.internal,}" Sep 11 18:15:17.376854 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.376835 2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jjfxh" Sep 11 18:15:17.383025 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:17.382947 2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-128.ec2.internal.18d456ea8db4945b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-128.ec2.internal,UID:ip-10-0-138-128.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-128.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-128.ec2.internal,},FirstTimestamp:2026-09-11 18:15:17.372687451 +0000 UTC m=+0.408626244,LastTimestamp:2026-09-11 18:15:17.372687451 +0000 UTC m=+0.408626244,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-128.ec2.internal,}" Sep 11 18:15:17.391044 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:17.390981 2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-128.ec2.internal.18d456ea8db4d5e1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-128.ec2.internal,UID:ip-10-0-138-128.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-138-128.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-138-128.ec2.internal,},FirstTimestamp:2026-09-11 18:15:17.372704225 +0000 UTC m=+0.408643021,LastTimestamp:2026-09-11 18:15:17.372704225 +0000 UTC m=+0.408643021,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-128.ec2.internal,}" Sep 11 18:15:17.402002 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:17.401931 2561 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-138-128.ec2.internal.18d456ea8db43d0d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-128.ec2.internal.18d456ea8db43d0d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-128.ec2.internal,UID:ip-10-0-138-128.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-128.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-128.ec2.internal,},FirstTimestamp:2026-09-11 18:15:17.372665101 +0000 UTC m=+0.408603899,LastTimestamp:2026-09-11 18:15:17.374526376 +0000 UTC m=+0.410465171,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-128.ec2.internal,}" Sep 11 18:15:17.516682 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:17.412901 2561 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-138-128.ec2.internal.18d456ea8db4945b\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-128.ec2.internal.18d456ea8db4945b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-128.ec2.internal,UID:ip-10-0-138-128.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-128.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-128.ec2.internal,},FirstTimestamp:2026-09-11 18:15:17.372687451 +0000 UTC m=+0.408626244,LastTimestamp:2026-09-11 18:15:17.374547856 +0000 UTC m=+0.410486651,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-128.ec2.internal,}" Sep 11 18:15:17.516682 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:17.423224 2561 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-138-128.ec2.internal.18d456ea8db4d5e1\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-128.ec2.internal.18d456ea8db4d5e1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-128.ec2.internal,UID:ip-10-0-138-128.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-138-128.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-138-128.ec2.internal,},FirstTimestamp:2026-09-11 18:15:17.372704225 +0000 UTC m=+0.408643021,LastTimestamp:2026-09-11 18:15:17.374560998 +0000 UTC m=+0.410499791,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-128.ec2.internal,}" Sep 11 18:15:17.516682 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:17.452390 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-128.ec2.internal\" not found" Sep 11 18:15:17.516958 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.516937 2561 policy_none.go:49] "None policy: Start" Sep 11 18:15:17.517014 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.516976 2561 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 11 18:15:17.517014 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.516990 2561 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 11 18:15:17.519204 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.519186 2561 policy_none.go:47] "Start" Sep 11 18:15:17.550240 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.550220 2561 manager.go:337] "Starting Device Plugin manager" Sep 11 18:15:17.550338 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:17.550266 2561 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 11 18:15:17.550338 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.550285 2561 server.go:87] "Starting device plugin registration server" Sep 11 18:15:17.550696 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.550683 2561 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 11 18:15:17.550736 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.550699 2561 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 11 18:15:17.550866 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.550849 2561 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 11 18:15:17.550937 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.550926 2561 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 11 18:15:17.550981 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.550939 2561 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 11 18:15:17.551655 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:17.551636 2561 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 11 18:15:17.551741 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:17.551691 2561 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-128.ec2.internal\" not found" Sep 11 18:15:17.573946 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:17.573914 2561 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-138-128.ec2.internal\" not found" node="ip-10-0-138-128.ec2.internal" Sep 11 18:15:17.591918 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.591884 2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 11 18:15:17.593194 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.593175 2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 11 18:15:17.593268 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.593244 2561 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 11 18:15:17.593318 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.593275 2561 kubelet.go:2448] "Starting kubelet main sync loop" Sep 11 18:15:17.593386 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:17.593368 2561 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 11 18:15:17.595954 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.595935 2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 18:15:17.651242 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.651187 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 18:15:17.652439 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.652411 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-128.ec2.internal" event="NodeHasSufficientMemory" Sep 11 18:15:17.652634 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.652451 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-128.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 18:15:17.652634 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.652467 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-128.ec2.internal" event="NodeHasSufficientPID" Sep 11 18:15:17.652634 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.652497 2561 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-128.ec2.internal" Sep 11 18:15:17.661971 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.661945 2561 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-128.ec2.internal" Sep 11 18:15:17.661971 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:17.661970 2561 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-128.ec2.internal\": node \"ip-10-0-138-128.ec2.internal\" not found" Sep 11 18:15:17.687398 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:17.687371 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-128.ec2.internal\" not found" Sep 11 18:15:17.694221 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.694196 2561 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-128.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-128.ec2.internal"] Sep 11 18:15:17.694292 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.694267 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 18:15:17.695174 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.695159 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-128.ec2.internal" event="NodeHasSufficientMemory" Sep 11 18:15:17.695271 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.695185 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-128.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 18:15:17.695271 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.695195 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-128.ec2.internal" event="NodeHasSufficientPID" Sep 11 18:15:17.697628 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.697614 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 18:15:17.697764 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.697750 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-128.ec2.internal" Sep 11 18:15:17.697798 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.697783 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 18:15:17.698344 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.698327 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-128.ec2.internal" event="NodeHasSufficientMemory" Sep 11 18:15:17.698344 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.698340 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-128.ec2.internal" event="NodeHasSufficientMemory" Sep 11 18:15:17.698446 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.698357 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-128.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 18:15:17.698446 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.698359 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-128.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 18:15:17.698446 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.698371 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-128.ec2.internal" event="NodeHasSufficientPID" Sep 11 18:15:17.698446 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.698374 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-128.ec2.internal" event="NodeHasSufficientPID" Sep 11 18:15:17.700670 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.700654 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-128.ec2.internal" Sep 11 18:15:17.700764 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.700677 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 18:15:17.701362 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.701341 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-128.ec2.internal" event="NodeHasSufficientMemory" Sep 11 18:15:17.701451 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.701391 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-128.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 18:15:17.701451 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.701402 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-128.ec2.internal" event="NodeHasSufficientPID" Sep 11 18:15:17.723846 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:17.723823 2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-128.ec2.internal\" not found" node="ip-10-0-138-128.ec2.internal" Sep 11 18:15:17.728274 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:17.728258 2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-128.ec2.internal\" not found" node="ip-10-0-138-128.ec2.internal" Sep 11 18:15:17.754934 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.754910 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ebf313d069610fccaa30accef842e07e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-128.ec2.internal\" (UID: \"ebf313d069610fccaa30accef842e07e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-128.ec2.internal" Sep 11 18:15:17.755021 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.754936 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/577bf0a7d48f3dd63acd7590896a721f-config\") pod \"kube-apiserver-proxy-ip-10-0-138-128.ec2.internal\" (UID: \"577bf0a7d48f3dd63acd7590896a721f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-128.ec2.internal" Sep 11 18:15:17.755021 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.754964 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ebf313d069610fccaa30accef842e07e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-128.ec2.internal\" (UID: \"ebf313d069610fccaa30accef842e07e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-128.ec2.internal" Sep 11 18:15:17.787596 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:17.787536 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-128.ec2.internal\" not found" Sep 11 18:15:17.856022 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.855991 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/577bf0a7d48f3dd63acd7590896a721f-config\") pod \"kube-apiserver-proxy-ip-10-0-138-128.ec2.internal\" (UID: \"577bf0a7d48f3dd63acd7590896a721f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-128.ec2.internal" Sep 11 18:15:17.856167 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.856047 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ebf313d069610fccaa30accef842e07e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-128.ec2.internal\" (UID: \"ebf313d069610fccaa30accef842e07e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-128.ec2.internal" Sep 11 18:15:17.856167 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.856063 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/577bf0a7d48f3dd63acd7590896a721f-config\") pod \"kube-apiserver-proxy-ip-10-0-138-128.ec2.internal\" (UID: \"577bf0a7d48f3dd63acd7590896a721f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-128.ec2.internal" Sep 11 18:15:17.856167 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.856075 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ebf313d069610fccaa30accef842e07e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-128.ec2.internal\" (UID: \"ebf313d069610fccaa30accef842e07e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-128.ec2.internal" Sep 11 18:15:17.856167 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.856116 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ebf313d069610fccaa30accef842e07e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-128.ec2.internal\" (UID: \"ebf313d069610fccaa30accef842e07e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-128.ec2.internal" Sep 11 18:15:17.856167 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:17.856127 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ebf313d069610fccaa30accef842e07e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-128.ec2.internal\" (UID: \"ebf313d069610fccaa30accef842e07e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-128.ec2.internal" Sep 11 18:15:17.888160 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:17.888116 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-128.ec2.internal\" not found" Sep 11 18:15:19.527629 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:19.527580 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-128.ec2.internal\" not found" Sep 11 18:15:19.564778 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:19.564749 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-128.ec2.internal" Sep 11 18:15:19.569691 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:19.569665 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-128.ec2.internal" Sep 11 18:15:19.627750 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:19.627671 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-128.ec2.internal\" not found" Sep 11 18:15:19.728129 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:19.728100 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-128.ec2.internal\" not found" Sep 11 18:15:19.781831 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:19.781808 2561 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 18:15:19.822151 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:19.822114 2561 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 11 18:15:19.822660 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:19.822282 2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 11 18:15:19.822660 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:19.822304 2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 11 18:15:19.828431 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:19.828406 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-128.ec2.internal\" not found" Sep 11 18:15:19.870385 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:19.870360 2561 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 18:15:19.891019 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:19.890946 2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-128.ec2.internal" Sep 11 18:15:19.891019 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:19.890966 2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 11 18:15:19.917138 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:19.917114 2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 11 18:15:19.919605 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:19.919586 2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-128.ec2.internal" Sep 11 18:15:19.922384 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:19.922364 2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 11 18:15:19.928012 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:19.927997 2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 11 18:15:20.030288 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.030251 2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-m445b" Sep 11 18:15:20.040017 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.039994 2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-m445b" Sep 11 18:15:20.060166 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.060120 2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 18:10:18 +0000 UTC" deadline="2028-05-25 02:51:30.596249256 +0000 UTC" Sep 11 18:15:20.060166 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.060162 2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14912h36m10.536089862s" Sep 11 18:15:20.181294 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:20.181251 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebf313d069610fccaa30accef842e07e.slice/crio-1ad65994cbaea40d1e21e70ac2402a41aca7e525e2d48afb372b870be6b280a1 WatchSource:0}: Error finding container 1ad65994cbaea40d1e21e70ac2402a41aca7e525e2d48afb372b870be6b280a1: Status 404 returned error can't find the container with id 1ad65994cbaea40d1e21e70ac2402a41aca7e525e2d48afb372b870be6b280a1 Sep 11 18:15:20.185558 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.185544 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 18:15:20.236293 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.236264 2561 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 18:15:20.402857 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:20.402815 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod577bf0a7d48f3dd63acd7590896a721f.slice/crio-e9ce59465e0215deac3ebd4d435129da65375f3671be53f66ec3980fefc49ef8 WatchSource:0}: Error finding container e9ce59465e0215deac3ebd4d435129da65375f3671be53f66ec3980fefc49ef8: Status 404 returned error can't find the container with id e9ce59465e0215deac3ebd4d435129da65375f3671be53f66ec3980fefc49ef8 Sep 11 18:15:20.866318 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.866289 2561 apiserver.go:52] "Watching apiserver" Sep 11 18:15:20.873690 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.873655 2561 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 11 18:15:20.874801 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.874775 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-76xpz","openshift-multus/network-metrics-daemon-cmf6c","openshift-network-diagnostics/network-check-target-zr7xc","openshift-network-operator/iptables-alerter-7b2vr","kube-system/konnectivity-agent-gt5ln","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fd24c","openshift-dns/node-resolver-xh28n","openshift-image-registry/node-ca-k7qjg","openshift-multus/multus-additional-cni-plugins-m9j9m","openshift-ovn-kubernetes/ovnkube-node-v4knd","kube-system/global-pull-secret-syncer-sm2rw","kube-system/kube-apiserver-proxy-ip-10-0-138-128.ec2.internal","openshift-cluster-node-tuning-operator/tuned-nwpdb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-128.ec2.internal"] Sep 11 18:15:20.877386 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.877364 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-76xpz" Sep 11 18:15:20.879598 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.879575 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cmf6c" Sep 11 18:15:20.879716 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:20.879664 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cmf6c" podUID="a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d" Sep 11 18:15:20.879995 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.879977 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-927s8\"" Sep 11 18:15:20.880243 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.880222 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 11 18:15:20.880320 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.880241 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 11 18:15:20.880320 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.880216 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 11 18:15:20.880425 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.880222 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 11 18:15:20.881787 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.881767 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr7xc" Sep 11 18:15:20.881894 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:20.881837 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr7xc" podUID="e183b6fd-207a-44ed-829a-6872d65d0a57" Sep 11 18:15:20.884013 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.883994 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7b2vr" Sep 11 18:15:20.886418 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.886399 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gt5ln" Sep 11 18:15:20.886733 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.886714 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-kqztf\"" Sep 11 18:15:20.886831 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.886810 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 11 18:15:20.886895 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.886713 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 11 18:15:20.886895 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.886881 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 11 18:15:20.888866 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.888503 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fd24c" Sep 11 18:15:20.889481 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.889220 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 11 18:15:20.889481 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.889282 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 11 18:15:20.889481 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.889311 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-mnd8f\"" Sep 11 18:15:20.891010 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.890672 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xh28n" Sep 11 18:15:20.891010 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.890705 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 11 18:15:20.891200 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.891181 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 11 18:15:20.891241 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.891183 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 11 18:15:20.891319 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.891298 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-v54qz\"" Sep 11 18:15:20.892881 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.892864 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-k7qjg" Sep 11 18:15:20.893128 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.893107 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 11 18:15:20.893219 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.893200 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 11 18:15:20.893404 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.893388 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9rcc8\"" Sep 11 18:15:20.895528 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.895505 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 11 18:15:20.895618 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.895547 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-m9j9m" Sep 11 18:15:20.895974 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.895945 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 11 18:15:20.896265 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.896090 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vm7kf\"" Sep 11 18:15:20.896265 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.896224 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 11 18:15:20.898452 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.898435 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-9dl6p\"" Sep 11 18:15:20.900133 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.898643 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:20.900133 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.899010 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 11 18:15:20.900133 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.899029 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 11 18:15:20.902496 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.902478 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 11 18:15:20.902850 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.902533 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm2rw" Sep 11 18:15:20.902992 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:20.902950 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm2rw" podUID="89de43e0-f41b-4570-af10-b3a3ef063480" Sep 11 18:15:20.903145 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.903109 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/afbcd3d2-b5c0-474c-a177-e42b915d310d-multus-cni-dir\") pod \"multus-76xpz\" (UID: \"afbcd3d2-b5c0-474c-a177-e42b915d310d\") " pod="openshift-multus/multus-76xpz" Sep 11 18:15:20.903231 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.903166 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6547cc6d-cb31-474a-8065-7fd47766fe02-host\") pod \"node-ca-k7qjg\" (UID: \"6547cc6d-cb31-474a-8065-7fd47766fe02\") " pod="openshift-image-registry/node-ca-k7qjg" Sep 11 18:15:20.903231 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.903218 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e9c4d02b-cc0d-4812-8fe2-80f5276923ba-agent-certs\") pod \"konnectivity-agent-gt5ln\" (UID: \"e9c4d02b-cc0d-4812-8fe2-80f5276923ba\") " pod="kube-system/konnectivity-agent-gt5ln" Sep 11 18:15:20.903333 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.903250 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/afbcd3d2-b5c0-474c-a177-e42b915d310d-cni-binary-copy\") pod \"multus-76xpz\" (UID: \"afbcd3d2-b5c0-474c-a177-e42b915d310d\") " pod="openshift-multus/multus-76xpz" Sep 11 18:15:20.903333 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.903276 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/afbcd3d2-b5c0-474c-a177-e42b915d310d-host-run-netns\") pod \"multus-76xpz\" (UID: \"afbcd3d2-b5c0-474c-a177-e42b915d310d\") " pod="openshift-multus/multus-76xpz" Sep 11 18:15:20.903333 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.903307 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/afbcd3d2-b5c0-474c-a177-e42b915d310d-host-var-lib-cni-multus\") pod \"multus-76xpz\" (UID: \"afbcd3d2-b5c0-474c-a177-e42b915d310d\") " pod="openshift-multus/multus-76xpz" Sep 11 18:15:20.903469 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.903341 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/afbcd3d2-b5c0-474c-a177-e42b915d310d-host-var-lib-kubelet\") pod \"multus-76xpz\" (UID: \"afbcd3d2-b5c0-474c-a177-e42b915d310d\") " pod="openshift-multus/multus-76xpz" Sep 11 18:15:20.903469 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.903372 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkls2\" (UniqueName: \"kubernetes.io/projected/afbcd3d2-b5c0-474c-a177-e42b915d310d-kube-api-access-hkls2\") pod \"multus-76xpz\" (UID: \"afbcd3d2-b5c0-474c-a177-e42b915d310d\") " pod="openshift-multus/multus-76xpz" Sep 11 18:15:20.903469 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.903426 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88qql\" (UniqueName: \"kubernetes.io/projected/6547cc6d-cb31-474a-8065-7fd47766fe02-kube-api-access-88qql\") pod \"node-ca-k7qjg\" (UID: \"6547cc6d-cb31-474a-8065-7fd47766fe02\") " pod="openshift-image-registry/node-ca-k7qjg" Sep 11 18:15:20.903617 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.903546 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 11 18:15:20.903617 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.903561 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f7d5659-22fb-43f1-8970-47837d126795-tmp-dir\") pod \"node-resolver-xh28n\" (UID: \"7f7d5659-22fb-43f1-8970-47837d126795\") " pod="openshift-dns/node-resolver-xh28n" Sep 11 18:15:20.903617 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.903606 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 11 18:15:20.903749 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.903629 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-znxc9\"" Sep 11 18:15:20.903749 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.903694 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/afbcd3d2-b5c0-474c-a177-e42b915d310d-cnibin\") pod \"multus-76xpz\" (UID: \"afbcd3d2-b5c0-474c-a177-e42b915d310d\") " pod="openshift-multus/multus-76xpz" Sep 11 18:15:20.903749 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.903737 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b7bfcee0-0fd4-40c3-bf79-50ac2a7abb80-host-slash\") pod \"iptables-alerter-7b2vr\" (UID: \"b7bfcee0-0fd4-40c3-bf79-50ac2a7abb80\") " pod="openshift-network-operator/iptables-alerter-7b2vr" Sep 11 18:15:20.903876 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.903777 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3c0fffdb-93cd-4936-96d6-1b4ef7d2486c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fd24c\" (UID: \"3c0fffdb-93cd-4936-96d6-1b4ef7d2486c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fd24c" Sep 11 18:15:20.903876 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.903853 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3c0fffdb-93cd-4936-96d6-1b4ef7d2486c-registration-dir\") pod \"aws-ebs-csi-driver-node-fd24c\" (UID: \"3c0fffdb-93cd-4936-96d6-1b4ef7d2486c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fd24c" Sep 11 18:15:20.903971 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.903878 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 11 18:15:20.903971 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.903891 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/afbcd3d2-b5c0-474c-a177-e42b915d310d-multus-conf-dir\") pod \"multus-76xpz\" (UID: \"afbcd3d2-b5c0-474c-a177-e42b915d310d\") " pod="openshift-multus/multus-76xpz" Sep 11 18:15:20.903971 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.903855 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 11 18:15:20.904137 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.903966 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 11 18:15:20.904137 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.903957 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5259\" (UniqueName: \"kubernetes.io/projected/7f7d5659-22fb-43f1-8970-47837d126795-kube-api-access-s5259\") pod \"node-resolver-xh28n\" (UID: \"7f7d5659-22fb-43f1-8970-47837d126795\") " pod="openshift-dns/node-resolver-xh28n" Sep 11 18:15:20.904137 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.904009 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/afbcd3d2-b5c0-474c-a177-e42b915d310d-host-var-lib-cni-bin\") pod \"multus-76xpz\" (UID: \"afbcd3d2-b5c0-474c-a177-e42b915d310d\") " pod="openshift-multus/multus-76xpz" Sep 11 18:15:20.904137 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.904119 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dq5x\" (UniqueName: \"kubernetes.io/projected/b7bfcee0-0fd4-40c3-bf79-50ac2a7abb80-kube-api-access-5dq5x\") pod \"iptables-alerter-7b2vr\" (UID: \"b7bfcee0-0fd4-40c3-bf79-50ac2a7abb80\") " pod="openshift-network-operator/iptables-alerter-7b2vr" Sep 11 18:15:20.904317 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.904155 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3c0fffdb-93cd-4936-96d6-1b4ef7d2486c-device-dir\") pod \"aws-ebs-csi-driver-node-fd24c\" (UID: \"3c0fffdb-93cd-4936-96d6-1b4ef7d2486c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fd24c" Sep 11 18:15:20.904317 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.904194 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3c0fffdb-93cd-4936-96d6-1b4ef7d2486c-etc-selinux\") pod \"aws-ebs-csi-driver-node-fd24c\" (UID: \"3c0fffdb-93cd-4936-96d6-1b4ef7d2486c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fd24c" Sep 11 18:15:20.904317 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.904270 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g28hv\" (UniqueName: \"kubernetes.io/projected/a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d-kube-api-access-g28hv\") pod \"network-metrics-daemon-cmf6c\" (UID: \"a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d\") " pod="openshift-multus/network-metrics-daemon-cmf6c" Sep 11 18:15:20.904317 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.904301 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e9c4d02b-cc0d-4812-8fe2-80f5276923ba-konnectivity-ca\") pod \"konnectivity-agent-gt5ln\" (UID: \"e9c4d02b-cc0d-4812-8fe2-80f5276923ba\") " pod="kube-system/konnectivity-agent-gt5ln" Sep 11 18:15:20.904513 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.904322 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/afbcd3d2-b5c0-474c-a177-e42b915d310d-multus-daemon-config\") pod \"multus-76xpz\" (UID: \"afbcd3d2-b5c0-474c-a177-e42b915d310d\") " pod="openshift-multus/multus-76xpz" Sep 11 18:15:20.904513 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.904380 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/afbcd3d2-b5c0-474c-a177-e42b915d310d-host-run-multus-certs\") pod \"multus-76xpz\" (UID: \"afbcd3d2-b5c0-474c-a177-e42b915d310d\") " pod="openshift-multus/multus-76xpz" Sep 11 18:15:20.904513 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.904458 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/afbcd3d2-b5c0-474c-a177-e42b915d310d-etc-kubernetes\") pod \"multus-76xpz\" (UID: \"afbcd3d2-b5c0-474c-a177-e42b915d310d\") " pod="openshift-multus/multus-76xpz" Sep 11 18:15:20.904659 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.904502 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g69tq\" (UniqueName: \"kubernetes.io/projected/e183b6fd-207a-44ed-829a-6872d65d0a57-kube-api-access-g69tq\") pod \"network-check-target-zr7xc\" (UID: \"e183b6fd-207a-44ed-829a-6872d65d0a57\") " pod="openshift-network-diagnostics/network-check-target-zr7xc" Sep 11 18:15:20.904659 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.904547 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3c0fffdb-93cd-4936-96d6-1b4ef7d2486c-sys-fs\") pod \"aws-ebs-csi-driver-node-fd24c\" (UID: \"3c0fffdb-93cd-4936-96d6-1b4ef7d2486c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fd24c" Sep 11 18:15:20.904659 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.904609 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7f7d5659-22fb-43f1-8970-47837d126795-hosts-file\") pod \"node-resolver-xh28n\" (UID: \"7f7d5659-22fb-43f1-8970-47837d126795\") " pod="openshift-dns/node-resolver-xh28n" Sep 11 18:15:20.904659 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.904642 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/afbcd3d2-b5c0-474c-a177-e42b915d310d-host-run-k8s-cni-cncf-io\") pod \"multus-76xpz\" (UID: \"afbcd3d2-b5c0-474c-a177-e42b915d310d\") " pod="openshift-multus/multus-76xpz" Sep 11 18:15:20.904840 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.904675 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b7bfcee0-0fd4-40c3-bf79-50ac2a7abb80-iptables-alerter-script\") pod \"iptables-alerter-7b2vr\" (UID: \"b7bfcee0-0fd4-40c3-bf79-50ac2a7abb80\") " pod="openshift-network-operator/iptables-alerter-7b2vr" Sep 11 18:15:20.904840 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.904725 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3c0fffdb-93cd-4936-96d6-1b4ef7d2486c-socket-dir\") pod \"aws-ebs-csi-driver-node-fd24c\" (UID: \"3c0fffdb-93cd-4936-96d6-1b4ef7d2486c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fd24c" Sep 11 18:15:20.904922 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.904889 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6547cc6d-cb31-474a-8065-7fd47766fe02-serviceca\") pod \"node-ca-k7qjg\" (UID: \"6547cc6d-cb31-474a-8065-7fd47766fe02\") " pod="openshift-image-registry/node-ca-k7qjg" Sep 11 18:15:20.904965 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.904938 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d-metrics-certs\") pod \"network-metrics-daemon-cmf6c\" (UID: \"a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d\") " pod="openshift-multus/network-metrics-daemon-cmf6c" Sep 11 18:15:20.905005 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.904979 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/afbcd3d2-b5c0-474c-a177-e42b915d310d-system-cni-dir\") pod \"multus-76xpz\" (UID: \"afbcd3d2-b5c0-474c-a177-e42b915d310d\") " pod="openshift-multus/multus-76xpz" Sep 11 18:15:20.905075 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.905017 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/afbcd3d2-b5c0-474c-a177-e42b915d310d-multus-socket-dir-parent\") pod \"multus-76xpz\" (UID: \"afbcd3d2-b5c0-474c-a177-e42b915d310d\") " pod="openshift-multus/multus-76xpz" Sep 11 18:15:20.905496 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.905476 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/afbcd3d2-b5c0-474c-a177-e42b915d310d-hostroot\") pod \"multus-76xpz\" (UID: \"afbcd3d2-b5c0-474c-a177-e42b915d310d\") " pod="openshift-multus/multus-76xpz" Sep 11 18:15:20.905575 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.905538 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zp8v\" (UniqueName: \"kubernetes.io/projected/3c0fffdb-93cd-4936-96d6-1b4ef7d2486c-kube-api-access-9zp8v\") pod \"aws-ebs-csi-driver-node-fd24c\" (UID: \"3c0fffdb-93cd-4936-96d6-1b4ef7d2486c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fd24c" Sep 11 18:15:20.905621 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.905603 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nwpdb" Sep 11 18:15:20.907956 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.907934 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 11 18:15:20.908049 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.907991 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 11 18:15:20.908167 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.908130 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fwv28\"" Sep 11 18:15:20.992478 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:20.992447 2561 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 11 18:15:21.005215 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.005183 2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 18:15:21.006317 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.006286 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/72de70ec-e2ce-47b2-883d-75f4eb8eb39c-sys\") pod \"tuned-nwpdb\" (UID: \"72de70ec-e2ce-47b2-883d-75f4eb8eb39c\") " pod="openshift-cluster-node-tuning-operator/tuned-nwpdb" Sep 11 18:15:21.006440 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.006330 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/72de70ec-e2ce-47b2-883d-75f4eb8eb39c-etc-tuned\") pod \"tuned-nwpdb\" (UID: \"72de70ec-e2ce-47b2-883d-75f4eb8eb39c\") " pod="openshift-cluster-node-tuning-operator/tuned-nwpdb" Sep 11 18:15:21.006440 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.006356 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/42e5ddd8-a96c-45a1-adf9-caa47ee902db-host-kubelet\") pod \"ovnkube-node-v4knd\" (UID: \"42e5ddd8-a96c-45a1-adf9-caa47ee902db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:21.006440 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.006406 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d-metrics-certs\") pod \"network-metrics-daemon-cmf6c\" (UID: \"a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d\") " pod="openshift-multus/network-metrics-daemon-cmf6c" Sep 11 18:15:21.006590 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.006451 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/afbcd3d2-b5c0-474c-a177-e42b915d310d-multus-cni-dir\") pod \"multus-76xpz\" (UID: \"afbcd3d2-b5c0-474c-a177-e42b915d310d\") " pod="openshift-multus/multus-76xpz" Sep 11 18:15:21.006590 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.006496 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/42e5ddd8-a96c-45a1-adf9-caa47ee902db-run-systemd\") pod \"ovnkube-node-v4knd\" (UID: \"42e5ddd8-a96c-45a1-adf9-caa47ee902db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:21.006590 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.006521 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/42e5ddd8-a96c-45a1-adf9-caa47ee902db-ovnkube-script-lib\") pod \"ovnkube-node-v4knd\" (UID: \"42e5ddd8-a96c-45a1-adf9-caa47ee902db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:21.006590 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:21.006527 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 18:15:21.006590 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.006572 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/afbcd3d2-b5c0-474c-a177-e42b915d310d-cni-binary-copy\") pod \"multus-76xpz\" (UID: \"afbcd3d2-b5c0-474c-a177-e42b915d310d\") " pod="openshift-multus/multus-76xpz" Sep 11 18:15:21.006830 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.006615 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/afbcd3d2-b5c0-474c-a177-e42b915d310d-multus-cni-dir\") pod \"multus-76xpz\" (UID: \"afbcd3d2-b5c0-474c-a177-e42b915d310d\") " pod="openshift-multus/multus-76xpz" Sep 11 18:15:21.006830 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:21.006624 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d-metrics-certs podName:a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d nodeName:}" failed. No retries permitted until 2026-09-11 18:15:21.506584416 +0000 UTC m=+3.003602161 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d-metrics-certs") pod "network-metrics-daemon-cmf6c" (UID: "a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 18:15:21.006830 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.006655 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/afbcd3d2-b5c0-474c-a177-e42b915d310d-host-var-lib-kubelet\") pod \"multus-76xpz\" (UID: \"afbcd3d2-b5c0-474c-a177-e42b915d310d\") " pod="openshift-multus/multus-76xpz" Sep 11 18:15:21.006830 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.006680 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/72de70ec-e2ce-47b2-883d-75f4eb8eb39c-lib-modules\") pod \"tuned-nwpdb\" (UID: \"72de70ec-e2ce-47b2-883d-75f4eb8eb39c\") " pod="openshift-cluster-node-tuning-operator/tuned-nwpdb" Sep 11 18:15:21.006830 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.006701 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/42e5ddd8-a96c-45a1-adf9-caa47ee902db-ovnkube-config\") pod \"ovnkube-node-v4knd\" (UID: \"42e5ddd8-a96c-45a1-adf9-caa47ee902db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:21.006830 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.006724 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/42e5ddd8-a96c-45a1-adf9-caa47ee902db-ovn-node-metrics-cert\") pod \"ovnkube-node-v4knd\" (UID: \"42e5ddd8-a96c-45a1-adf9-caa47ee902db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:21.006830 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.006747 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/89de43e0-f41b-4570-af10-b3a3ef063480-dbus\") pod \"global-pull-secret-syncer-sm2rw\" (UID: \"89de43e0-f41b-4570-af10-b3a3ef063480\") " pod="kube-system/global-pull-secret-syncer-sm2rw" Sep 11 18:15:21.006830 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.006773 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f7d5659-22fb-43f1-8970-47837d126795-tmp-dir\") pod \"node-resolver-xh28n\" (UID: \"7f7d5659-22fb-43f1-8970-47837d126795\") " pod="openshift-dns/node-resolver-xh28n" Sep 11 18:15:21.006830 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.006781 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/afbcd3d2-b5c0-474c-a177-e42b915d310d-host-var-lib-kubelet\") pod \"multus-76xpz\" (UID: \"afbcd3d2-b5c0-474c-a177-e42b915d310d\") " pod="openshift-multus/multus-76xpz" Sep 11 18:15:21.006830 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.006813 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/72de70ec-e2ce-47b2-883d-75f4eb8eb39c-run\") pod \"tuned-nwpdb\" (UID: \"72de70ec-e2ce-47b2-883d-75f4eb8eb39c\") " pod="openshift-cluster-node-tuning-operator/tuned-nwpdb" Sep 11 18:15:21.007210 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.006866 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl2fr\" (UniqueName: \"kubernetes.io/projected/4d1be3a0-f7d6-475b-b79f-c4fde3f4bcb9-kube-api-access-sl2fr\") pod \"multus-additional-cni-plugins-m9j9m\" (UID: \"4d1be3a0-f7d6-475b-b79f-c4fde3f4bcb9\") " pod="openshift-multus/multus-additional-cni-plugins-m9j9m" Sep 11 18:15:21.007210 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.006911 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/afbcd3d2-b5c0-474c-a177-e42b915d310d-multus-conf-dir\") pod \"multus-76xpz\" (UID: \"afbcd3d2-b5c0-474c-a177-e42b915d310d\") " pod="openshift-multus/multus-76xpz" Sep 11 18:15:21.007210 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.006935 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/72de70ec-e2ce-47b2-883d-75f4eb8eb39c-etc-modprobe-d\") pod \"tuned-nwpdb\" (UID: \"72de70ec-e2ce-47b2-883d-75f4eb8eb39c\") " pod="openshift-cluster-node-tuning-operator/tuned-nwpdb" Sep 11 18:15:21.007210 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.006958 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj82v\" (UniqueName: \"kubernetes.io/projected/72de70ec-e2ce-47b2-883d-75f4eb8eb39c-kube-api-access-sj82v\") pod \"tuned-nwpdb\" (UID: \"72de70ec-e2ce-47b2-883d-75f4eb8eb39c\") " pod="openshift-cluster-node-tuning-operator/tuned-nwpdb" Sep 11 18:15:21.007210 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.006981 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d1be3a0-f7d6-475b-b79f-c4fde3f4bcb9-system-cni-dir\") pod \"multus-additional-cni-plugins-m9j9m\" (UID: \"4d1be3a0-f7d6-475b-b79f-c4fde3f4bcb9\") " pod="openshift-multus/multus-additional-cni-plugins-m9j9m" Sep 11 18:15:21.007210 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.007003 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/afbcd3d2-b5c0-474c-a177-e42b915d310d-multus-conf-dir\") pod \"multus-76xpz\" (UID: \"afbcd3d2-b5c0-474c-a177-e42b915d310d\") " pod="openshift-multus/multus-76xpz" Sep 11 18:15:21.007210 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.007006 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5259\" (UniqueName: \"kubernetes.io/projected/7f7d5659-22fb-43f1-8970-47837d126795-kube-api-access-s5259\") pod \"node-resolver-xh28n\" (UID: \"7f7d5659-22fb-43f1-8970-47837d126795\") " pod="openshift-dns/node-resolver-xh28n" Sep 11 18:15:21.007210 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.007068 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dq5x\" (UniqueName: \"kubernetes.io/projected/b7bfcee0-0fd4-40c3-bf79-50ac2a7abb80-kube-api-access-5dq5x\") pod \"iptables-alerter-7b2vr\" (UID: \"b7bfcee0-0fd4-40c3-bf79-50ac2a7abb80\") " pod="openshift-network-operator/iptables-alerter-7b2vr" Sep 11 18:15:21.007210 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.007093 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3c0fffdb-93cd-4936-96d6-1b4ef7d2486c-device-dir\") pod \"aws-ebs-csi-driver-node-fd24c\" (UID: \"3c0fffdb-93cd-4936-96d6-1b4ef7d2486c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fd24c" Sep 11 18:15:21.007210 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.007111 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3c0fffdb-93cd-4936-96d6-1b4ef7d2486c-etc-selinux\") pod \"aws-ebs-csi-driver-node-fd24c\" (UID: \"3c0fffdb-93cd-4936-96d6-1b4ef7d2486c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fd24c" Sep 11 18:15:21.007210 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.007127 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72de70ec-e2ce-47b2-883d-75f4eb8eb39c-etc-kubernetes\") pod \"tuned-nwpdb\" (UID: \"72de70ec-e2ce-47b2-883d-75f4eb8eb39c\") " pod="openshift-cluster-node-tuning-operator/tuned-nwpdb" Sep 11 18:15:21.007210 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.007142 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/42e5ddd8-a96c-45a1-adf9-caa47ee902db-systemd-units\") pod \"ovnkube-node-v4knd\" (UID: \"42e5ddd8-a96c-45a1-adf9-caa47ee902db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:21.007210 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.007144 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f7d5659-22fb-43f1-8970-47837d126795-tmp-dir\") pod \"node-resolver-xh28n\" (UID: \"7f7d5659-22fb-43f1-8970-47837d126795\") " pod="openshift-dns/node-resolver-xh28n" Sep 11 18:15:21.007210 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.007157 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/42e5ddd8-a96c-45a1-adf9-caa47ee902db-node-log\") pod \"ovnkube-node-v4knd\" (UID: \"42e5ddd8-a96c-45a1-adf9-caa47ee902db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:21.007621 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.007232 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g28hv\" (UniqueName: \"kubernetes.io/projected/a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d-kube-api-access-g28hv\") pod \"network-metrics-daemon-cmf6c\" (UID: \"a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d\") " pod="openshift-multus/network-metrics-daemon-cmf6c" Sep 11 18:15:21.007621 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.007247 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/afbcd3d2-b5c0-474c-a177-e42b915d310d-cni-binary-copy\") pod \"multus-76xpz\" (UID: \"afbcd3d2-b5c0-474c-a177-e42b915d310d\") " pod="openshift-multus/multus-76xpz" Sep 11 18:15:21.007621 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.007265 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e9c4d02b-cc0d-4812-8fe2-80f5276923ba-konnectivity-ca\") pod \"konnectivity-agent-gt5ln\" (UID: \"e9c4d02b-cc0d-4812-8fe2-80f5276923ba\") " pod="kube-system/konnectivity-agent-gt5ln" Sep 11 18:15:21.007621 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.007295 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/afbcd3d2-b5c0-474c-a177-e42b915d310d-etc-kubernetes\") pod \"multus-76xpz\" (UID: \"afbcd3d2-b5c0-474c-a177-e42b915d310d\") " pod="openshift-multus/multus-76xpz" Sep 11 18:15:21.007621 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.007323 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g69tq\" (UniqueName: \"kubernetes.io/projected/e183b6fd-207a-44ed-829a-6872d65d0a57-kube-api-access-g69tq\") pod \"network-check-target-zr7xc\" (UID: \"e183b6fd-207a-44ed-829a-6872d65d0a57\") " pod="openshift-network-diagnostics/network-check-target-zr7xc" Sep 11 18:15:21.007621 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.007359 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3c0fffdb-93cd-4936-96d6-1b4ef7d2486c-device-dir\") pod \"aws-ebs-csi-driver-node-fd24c\" (UID: \"3c0fffdb-93cd-4936-96d6-1b4ef7d2486c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fd24c" Sep 11 18:15:21.007621 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.007363 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3c0fffdb-93cd-4936-96d6-1b4ef7d2486c-sys-fs\") pod \"aws-ebs-csi-driver-node-fd24c\" (UID: \"3c0fffdb-93cd-4936-96d6-1b4ef7d2486c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fd24c" Sep 11 18:15:21.007621 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.007393 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/afbcd3d2-b5c0-474c-a177-e42b915d310d-etc-kubernetes\") pod \"multus-76xpz\" (UID: \"afbcd3d2-b5c0-474c-a177-e42b915d310d\") " pod="openshift-multus/multus-76xpz" Sep 11 18:15:21.007621 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.007402 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3c0fffdb-93cd-4936-96d6-1b4ef7d2486c-etc-selinux\") pod \"aws-ebs-csi-driver-node-fd24c\" (UID: \"3c0fffdb-93cd-4936-96d6-1b4ef7d2486c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fd24c" Sep 11 18:15:21.007621 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.007408 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/42e5ddd8-a96c-45a1-adf9-caa47ee902db-host-slash\") pod \"ovnkube-node-v4knd\" (UID: \"42e5ddd8-a96c-45a1-adf9-caa47ee902db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:21.007621 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.007436 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/42e5ddd8-a96c-45a1-adf9-caa47ee902db-host-run-netns\") pod \"ovnkube-node-v4knd\" (UID: \"42e5ddd8-a96c-45a1-adf9-caa47ee902db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:21.007621 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.007502 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3c0fffdb-93cd-4936-96d6-1b4ef7d2486c-sys-fs\") pod \"aws-ebs-csi-driver-node-fd24c\" (UID: \"3c0fffdb-93cd-4936-96d6-1b4ef7d2486c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fd24c" Sep 11 18:15:21.007621 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.007543 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/42e5ddd8-a96c-45a1-adf9-caa47ee902db-run-ovn\") pod \"ovnkube-node-v4knd\" (UID: \"42e5ddd8-a96c-45a1-adf9-caa47ee902db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:21.008228 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.007670 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/afbcd3d2-b5c0-474c-a177-e42b915d310d-host-run-k8s-cni-cncf-io\") pod \"multus-76xpz\" (UID: \"afbcd3d2-b5c0-474c-a177-e42b915d310d\") " pod="openshift-multus/multus-76xpz" Sep 11 18:15:21.008228 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.007714 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/afbcd3d2-b5c0-474c-a177-e42b915d310d-host-run-k8s-cni-cncf-io\") pod \"multus-76xpz\" (UID: \"afbcd3d2-b5c0-474c-a177-e42b915d310d\") " pod="openshift-multus/multus-76xpz" Sep 11 18:15:21.008228 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.007750 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b7bfcee0-0fd4-40c3-bf79-50ac2a7abb80-iptables-alerter-script\") pod \"iptables-alerter-7b2vr\" (UID: \"b7bfcee0-0fd4-40c3-bf79-50ac2a7abb80\") " pod="openshift-network-operator/iptables-alerter-7b2vr" Sep 11 18:15:21.008228 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.007802 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3c0fffdb-93cd-4936-96d6-1b4ef7d2486c-socket-dir\") pod \"aws-ebs-csi-driver-node-fd24c\" (UID: \"3c0fffdb-93cd-4936-96d6-1b4ef7d2486c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fd24c" Sep 11 18:15:21.008228 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.007862 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6547cc6d-cb31-474a-8065-7fd47766fe02-serviceca\") pod \"node-ca-k7qjg\" (UID: \"6547cc6d-cb31-474a-8065-7fd47766fe02\") " pod="openshift-image-registry/node-ca-k7qjg" Sep 11 18:15:21.008228 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.007883 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e9c4d02b-cc0d-4812-8fe2-80f5276923ba-konnectivity-ca\") pod \"konnectivity-agent-gt5ln\" (UID: \"e9c4d02b-cc0d-4812-8fe2-80f5276923ba\") " pod="kube-system/konnectivity-agent-gt5ln" Sep 11 18:15:21.008228 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.007917 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/72de70ec-e2ce-47b2-883d-75f4eb8eb39c-tmp\") pod \"tuned-nwpdb\" (UID: \"72de70ec-e2ce-47b2-883d-75f4eb8eb39c\") " pod="openshift-cluster-node-tuning-operator/tuned-nwpdb" Sep 11 18:15:21.008228 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.007977 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3c0fffdb-93cd-4936-96d6-1b4ef7d2486c-socket-dir\") pod \"aws-ebs-csi-driver-node-fd24c\" (UID: \"3c0fffdb-93cd-4936-96d6-1b4ef7d2486c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fd24c" Sep 11 18:15:21.008228 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.008027 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/42e5ddd8-a96c-45a1-adf9-caa47ee902db-etc-openvswitch\") pod \"ovnkube-node-v4knd\" (UID: \"42e5ddd8-a96c-45a1-adf9-caa47ee902db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:21.008228 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.008074 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnwww\" (UniqueName: \"kubernetes.io/projected/42e5ddd8-a96c-45a1-adf9-caa47ee902db-kube-api-access-cnwww\") pod \"ovnkube-node-v4knd\" (UID: \"42e5ddd8-a96c-45a1-adf9-caa47ee902db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:21.008228 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.008126 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/89de43e0-f41b-4570-af10-b3a3ef063480-original-pull-secret\") pod \"global-pull-secret-syncer-sm2rw\" (UID: \"89de43e0-f41b-4570-af10-b3a3ef063480\") " pod="kube-system/global-pull-secret-syncer-sm2rw" Sep 11 18:15:21.008228 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.008165 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/afbcd3d2-b5c0-474c-a177-e42b915d310d-system-cni-dir\") pod \"multus-76xpz\" (UID: \"afbcd3d2-b5c0-474c-a177-e42b915d310d\") " pod="openshift-multus/multus-76xpz" Sep 11 18:15:21.008228 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.008192 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/afbcd3d2-b5c0-474c-a177-e42b915d310d-multus-socket-dir-parent\") pod \"multus-76xpz\" (UID: \"afbcd3d2-b5c0-474c-a177-e42b915d310d\") " pod="openshift-multus/multus-76xpz" Sep 11 18:15:21.008228 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.008218 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/afbcd3d2-b5c0-474c-a177-e42b915d310d-hostroot\") pod \"multus-76xpz\" (UID: \"afbcd3d2-b5c0-474c-a177-e42b915d310d\") " pod="openshift-multus/multus-76xpz" Sep 11 18:15:21.008887 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.008247 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zp8v\" (UniqueName: \"kubernetes.io/projected/3c0fffdb-93cd-4936-96d6-1b4ef7d2486c-kube-api-access-9zp8v\") pod \"aws-ebs-csi-driver-node-fd24c\" (UID: \"3c0fffdb-93cd-4936-96d6-1b4ef7d2486c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fd24c" Sep 11 18:15:21.008887 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.008292 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/72de70ec-e2ce-47b2-883d-75f4eb8eb39c-etc-sysconfig\") pod \"tuned-nwpdb\" (UID: \"72de70ec-e2ce-47b2-883d-75f4eb8eb39c\") " pod="openshift-cluster-node-tuning-operator/tuned-nwpdb" Sep 11 18:15:21.008887 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.008322 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/72de70ec-e2ce-47b2-883d-75f4eb8eb39c-etc-sysctl-conf\") pod \"tuned-nwpdb\" (UID: \"72de70ec-e2ce-47b2-883d-75f4eb8eb39c\") " pod="openshift-cluster-node-tuning-operator/tuned-nwpdb" Sep 11 18:15:21.008887 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.008294 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6547cc6d-cb31-474a-8065-7fd47766fe02-serviceca\") pod \"node-ca-k7qjg\" (UID: \"6547cc6d-cb31-474a-8065-7fd47766fe02\") " pod="openshift-image-registry/node-ca-k7qjg" Sep 11 18:15:21.008887 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.008325 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/afbcd3d2-b5c0-474c-a177-e42b915d310d-system-cni-dir\") pod \"multus-76xpz\" (UID: \"afbcd3d2-b5c0-474c-a177-e42b915d310d\") " pod="openshift-multus/multus-76xpz" Sep 11 18:15:21.008887 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.008353 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/72de70ec-e2ce-47b2-883d-75f4eb8eb39c-var-lib-kubelet\") pod \"tuned-nwpdb\" (UID: \"72de70ec-e2ce-47b2-883d-75f4eb8eb39c\") " pod="openshift-cluster-node-tuning-operator/tuned-nwpdb" Sep 11 18:15:21.008887 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.008324 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b7bfcee0-0fd4-40c3-bf79-50ac2a7abb80-iptables-alerter-script\") pod \"iptables-alerter-7b2vr\" (UID: \"b7bfcee0-0fd4-40c3-bf79-50ac2a7abb80\") " pod="openshift-network-operator/iptables-alerter-7b2vr" Sep 11 18:15:21.008887 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.008386 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4d1be3a0-f7d6-475b-b79f-c4fde3f4bcb9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m9j9m\" (UID: \"4d1be3a0-f7d6-475b-b79f-c4fde3f4bcb9\") " pod="openshift-multus/multus-additional-cni-plugins-m9j9m" Sep 11 18:15:21.008887 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.008401 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/afbcd3d2-b5c0-474c-a177-e42b915d310d-hostroot\") pod \"multus-76xpz\" (UID: \"afbcd3d2-b5c0-474c-a177-e42b915d310d\") " pod="openshift-multus/multus-76xpz" Sep 11 18:15:21.008887 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.008413 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6547cc6d-cb31-474a-8065-7fd47766fe02-host\") pod \"node-ca-k7qjg\" (UID: \"6547cc6d-cb31-474a-8065-7fd47766fe02\") " pod="openshift-image-registry/node-ca-k7qjg" Sep 11 18:15:21.008887 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.008453 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/42e5ddd8-a96c-45a1-adf9-caa47ee902db-host-cni-netd\") pod \"ovnkube-node-v4knd\" (UID: \"42e5ddd8-a96c-45a1-adf9-caa47ee902db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:21.008887 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.008458 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6547cc6d-cb31-474a-8065-7fd47766fe02-host\") pod \"node-ca-k7qjg\" (UID: \"6547cc6d-cb31-474a-8065-7fd47766fe02\") " pod="openshift-image-registry/node-ca-k7qjg" Sep 11 18:15:21.008887 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.008481 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e9c4d02b-cc0d-4812-8fe2-80f5276923ba-agent-certs\") pod \"konnectivity-agent-gt5ln\" (UID: \"e9c4d02b-cc0d-4812-8fe2-80f5276923ba\") " pod="kube-system/konnectivity-agent-gt5ln" Sep 11 18:15:21.008887 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.008520 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/afbcd3d2-b5c0-474c-a177-e42b915d310d-host-run-netns\") pod \"multus-76xpz\" (UID: \"afbcd3d2-b5c0-474c-a177-e42b915d310d\") " pod="openshift-multus/multus-76xpz" Sep 11 18:15:21.008887 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.008481 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/afbcd3d2-b5c0-474c-a177-e42b915d310d-multus-socket-dir-parent\") pod \"multus-76xpz\" (UID: \"afbcd3d2-b5c0-474c-a177-e42b915d310d\") " pod="openshift-multus/multus-76xpz" Sep 11 18:15:21.008887 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.008553 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/afbcd3d2-b5c0-474c-a177-e42b915d310d-host-var-lib-cni-multus\") pod \"multus-76xpz\" (UID: \"afbcd3d2-b5c0-474c-a177-e42b915d310d\") " pod="openshift-multus/multus-76xpz" Sep 11 18:15:21.008887 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.008571 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/afbcd3d2-b5c0-474c-a177-e42b915d310d-host-run-netns\") pod \"multus-76xpz\" (UID: \"afbcd3d2-b5c0-474c-a177-e42b915d310d\") " pod="openshift-multus/multus-76xpz" Sep 11 18:15:21.009704 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.008594 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkls2\" (UniqueName: \"kubernetes.io/projected/afbcd3d2-b5c0-474c-a177-e42b915d310d-kube-api-access-hkls2\") pod \"multus-76xpz\" (UID: \"afbcd3d2-b5c0-474c-a177-e42b915d310d\") " pod="openshift-multus/multus-76xpz" Sep 11 18:15:21.009704 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.008626 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88qql\" (UniqueName: \"kubernetes.io/projected/6547cc6d-cb31-474a-8065-7fd47766fe02-kube-api-access-88qql\") pod \"node-ca-k7qjg\" (UID: \"6547cc6d-cb31-474a-8065-7fd47766fe02\") " pod="openshift-image-registry/node-ca-k7qjg" Sep 11 18:15:21.009704 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.008655 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4d1be3a0-f7d6-475b-b79f-c4fde3f4bcb9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m9j9m\" (UID: \"4d1be3a0-f7d6-475b-b79f-c4fde3f4bcb9\") " pod="openshift-multus/multus-additional-cni-plugins-m9j9m" Sep 11 18:15:21.009704 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.008681 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/42e5ddd8-a96c-45a1-adf9-caa47ee902db-run-openvswitch\") pod \"ovnkube-node-v4knd\" (UID: \"42e5ddd8-a96c-45a1-adf9-caa47ee902db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:21.009704 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.008698 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/afbcd3d2-b5c0-474c-a177-e42b915d310d-cnibin\") pod \"multus-76xpz\" (UID: \"afbcd3d2-b5c0-474c-a177-e42b915d310d\") " pod="openshift-multus/multus-76xpz" Sep 11 18:15:21.009704 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.008726 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b7bfcee0-0fd4-40c3-bf79-50ac2a7abb80-host-slash\") pod \"iptables-alerter-7b2vr\" (UID: \"b7bfcee0-0fd4-40c3-bf79-50ac2a7abb80\") " pod="openshift-network-operator/iptables-alerter-7b2vr" Sep 11 18:15:21.009704 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.008741 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3c0fffdb-93cd-4936-96d6-1b4ef7d2486c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fd24c\" (UID: \"3c0fffdb-93cd-4936-96d6-1b4ef7d2486c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fd24c" Sep 11 18:15:21.009704 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.008757 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3c0fffdb-93cd-4936-96d6-1b4ef7d2486c-registration-dir\") pod \"aws-ebs-csi-driver-node-fd24c\" (UID: \"3c0fffdb-93cd-4936-96d6-1b4ef7d2486c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fd24c" Sep 11 18:15:21.009704 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.008788 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/afbcd3d2-b5c0-474c-a177-e42b915d310d-host-var-lib-cni-multus\") pod \"multus-76xpz\" (UID: \"afbcd3d2-b5c0-474c-a177-e42b915d310d\") " pod="openshift-multus/multus-76xpz" Sep 11 18:15:21.009704 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.008800 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/72de70ec-e2ce-47b2-883d-75f4eb8eb39c-etc-sysctl-d\") pod \"tuned-nwpdb\" (UID: \"72de70ec-e2ce-47b2-883d-75f4eb8eb39c\") " pod="openshift-cluster-node-tuning-operator/tuned-nwpdb" Sep 11 18:15:21.009704 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.008827 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4d1be3a0-f7d6-475b-b79f-c4fde3f4bcb9-cnibin\") pod \"multus-additional-cni-plugins-m9j9m\" (UID: \"4d1be3a0-f7d6-475b-b79f-c4fde3f4bcb9\") " pod="openshift-multus/multus-additional-cni-plugins-m9j9m" Sep 11 18:15:21.009704 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.008851 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4d1be3a0-f7d6-475b-b79f-c4fde3f4bcb9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m9j9m\" (UID: \"4d1be3a0-f7d6-475b-b79f-c4fde3f4bcb9\") " pod="openshift-multus/multus-additional-cni-plugins-m9j9m" Sep 11 18:15:21.009704 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.008875 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/72de70ec-e2ce-47b2-883d-75f4eb8eb39c-etc-systemd\") pod \"tuned-nwpdb\" (UID: \"72de70ec-e2ce-47b2-883d-75f4eb8eb39c\") " pod="openshift-cluster-node-tuning-operator/tuned-nwpdb" Sep 11 18:15:21.009704 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.008898 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/42e5ddd8-a96c-45a1-adf9-caa47ee902db-log-socket\") pod \"ovnkube-node-v4knd\" (UID: \"42e5ddd8-a96c-45a1-adf9-caa47ee902db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:21.009704 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.008926 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/42e5ddd8-a96c-45a1-adf9-caa47ee902db-host-run-ovn-kubernetes\") pod \"ovnkube-node-v4knd\" (UID: \"42e5ddd8-a96c-45a1-adf9-caa47ee902db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:21.009704 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.008963 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/42e5ddd8-a96c-45a1-adf9-caa47ee902db-host-cni-bin\") pod \"ovnkube-node-v4knd\" (UID: \"42e5ddd8-a96c-45a1-adf9-caa47ee902db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:21.010326 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.008990 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/42e5ddd8-a96c-45a1-adf9-caa47ee902db-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v4knd\" (UID: \"42e5ddd8-a96c-45a1-adf9-caa47ee902db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:21.010326 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.008980 2561 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 11 18:15:21.010326 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.009020 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/89de43e0-f41b-4570-af10-b3a3ef063480-kubelet-config\") pod \"global-pull-secret-syncer-sm2rw\" (UID: \"89de43e0-f41b-4570-af10-b3a3ef063480\") " pod="kube-system/global-pull-secret-syncer-sm2rw" Sep 11 18:15:21.010326 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.009038 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3c0fffdb-93cd-4936-96d6-1b4ef7d2486c-registration-dir\") pod \"aws-ebs-csi-driver-node-fd24c\" (UID: \"3c0fffdb-93cd-4936-96d6-1b4ef7d2486c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fd24c" Sep 11 18:15:21.010326 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.009044 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/afbcd3d2-b5c0-474c-a177-e42b915d310d-host-var-lib-cni-bin\") pod \"multus-76xpz\" (UID: \"afbcd3d2-b5c0-474c-a177-e42b915d310d\") " pod="openshift-multus/multus-76xpz" Sep 11 18:15:21.010326 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.009098 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/42e5ddd8-a96c-45a1-adf9-caa47ee902db-var-lib-openvswitch\") pod \"ovnkube-node-v4knd\" (UID: \"42e5ddd8-a96c-45a1-adf9-caa47ee902db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:21.010326 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.009101 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/afbcd3d2-b5c0-474c-a177-e42b915d310d-host-var-lib-cni-bin\") pod \"multus-76xpz\" (UID: \"afbcd3d2-b5c0-474c-a177-e42b915d310d\") " pod="openshift-multus/multus-76xpz" Sep 11 18:15:21.010326 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.009141 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/42e5ddd8-a96c-45a1-adf9-caa47ee902db-env-overrides\") pod \"ovnkube-node-v4knd\" (UID: \"42e5ddd8-a96c-45a1-adf9-caa47ee902db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:21.010326 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.009300 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/afbcd3d2-b5c0-474c-a177-e42b915d310d-cnibin\") pod \"multus-76xpz\" (UID: \"afbcd3d2-b5c0-474c-a177-e42b915d310d\") " pod="openshift-multus/multus-76xpz" Sep 11 18:15:21.010326 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.009341 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b7bfcee0-0fd4-40c3-bf79-50ac2a7abb80-host-slash\") pod \"iptables-alerter-7b2vr\" (UID: \"b7bfcee0-0fd4-40c3-bf79-50ac2a7abb80\") " pod="openshift-network-operator/iptables-alerter-7b2vr" Sep 11 18:15:21.010326 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.009395 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3c0fffdb-93cd-4936-96d6-1b4ef7d2486c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fd24c\" (UID: \"3c0fffdb-93cd-4936-96d6-1b4ef7d2486c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fd24c" Sep 11 18:15:21.010326 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.009432 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/afbcd3d2-b5c0-474c-a177-e42b915d310d-multus-daemon-config\") pod \"multus-76xpz\" (UID: \"afbcd3d2-b5c0-474c-a177-e42b915d310d\") " pod="openshift-multus/multus-76xpz" Sep 11 18:15:21.010326 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.009548 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/afbcd3d2-b5c0-474c-a177-e42b915d310d-host-run-multus-certs\") pod \"multus-76xpz\" (UID: \"afbcd3d2-b5c0-474c-a177-e42b915d310d\") " pod="openshift-multus/multus-76xpz" Sep 11 18:15:21.010326 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.009626 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/72de70ec-e2ce-47b2-883d-75f4eb8eb39c-host\") pod \"tuned-nwpdb\" (UID: \"72de70ec-e2ce-47b2-883d-75f4eb8eb39c\") " pod="openshift-cluster-node-tuning-operator/tuned-nwpdb" Sep 11 18:15:21.010326 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.009655 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4d1be3a0-f7d6-475b-b79f-c4fde3f4bcb9-cni-binary-copy\") pod \"multus-additional-cni-plugins-m9j9m\" (UID: \"4d1be3a0-f7d6-475b-b79f-c4fde3f4bcb9\") " pod="openshift-multus/multus-additional-cni-plugins-m9j9m" Sep 11 18:15:21.010326 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.009690 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7f7d5659-22fb-43f1-8970-47837d126795-hosts-file\") pod \"node-resolver-xh28n\" (UID: \"7f7d5659-22fb-43f1-8970-47837d126795\") " pod="openshift-dns/node-resolver-xh28n" Sep 11 18:15:21.010326 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.009771 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7f7d5659-22fb-43f1-8970-47837d126795-hosts-file\") pod \"node-resolver-xh28n\" (UID: \"7f7d5659-22fb-43f1-8970-47837d126795\") " pod="openshift-dns/node-resolver-xh28n" Sep 11 18:15:21.011178 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.009807 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/afbcd3d2-b5c0-474c-a177-e42b915d310d-host-run-multus-certs\") pod \"multus-76xpz\" (UID: \"afbcd3d2-b5c0-474c-a177-e42b915d310d\") " pod="openshift-multus/multus-76xpz" Sep 11 18:15:21.011178 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.009854 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/afbcd3d2-b5c0-474c-a177-e42b915d310d-multus-daemon-config\") pod \"multus-76xpz\" (UID: \"afbcd3d2-b5c0-474c-a177-e42b915d310d\") " pod="openshift-multus/multus-76xpz" Sep 11 18:15:21.013761 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.013741 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e9c4d02b-cc0d-4812-8fe2-80f5276923ba-agent-certs\") pod \"konnectivity-agent-gt5ln\" (UID: \"e9c4d02b-cc0d-4812-8fe2-80f5276923ba\") " pod="kube-system/konnectivity-agent-gt5ln" Sep 11 18:15:21.016124 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:21.016081 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 18:15:21.016124 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:21.016108 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 18:15:21.016124 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:21.016122 2561 projected.go:196] Error preparing data for projected volume kube-api-access-g69tq for pod openshift-network-diagnostics/network-check-target-zr7xc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 18:15:21.016325 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:21.016181 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e183b6fd-207a-44ed-829a-6872d65d0a57-kube-api-access-g69tq podName:e183b6fd-207a-44ed-829a-6872d65d0a57 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:21.516163877 +0000 UTC m=+3.013181631 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-g69tq" (UniqueName: "kubernetes.io/projected/e183b6fd-207a-44ed-829a-6872d65d0a57-kube-api-access-g69tq") pod "network-check-target-zr7xc" (UID: "e183b6fd-207a-44ed-829a-6872d65d0a57") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 18:15:21.019009 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.018951 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5259\" (UniqueName: \"kubernetes.io/projected/7f7d5659-22fb-43f1-8970-47837d126795-kube-api-access-s5259\") pod \"node-resolver-xh28n\" (UID: \"7f7d5659-22fb-43f1-8970-47837d126795\") " pod="openshift-dns/node-resolver-xh28n" Sep 11 18:15:21.020904 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.020884 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88qql\" (UniqueName: \"kubernetes.io/projected/6547cc6d-cb31-474a-8065-7fd47766fe02-kube-api-access-88qql\") pod \"node-ca-k7qjg\" (UID: \"6547cc6d-cb31-474a-8065-7fd47766fe02\") " pod="openshift-image-registry/node-ca-k7qjg" Sep 11 18:15:21.021004 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.020968 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dq5x\" (UniqueName: \"kubernetes.io/projected/b7bfcee0-0fd4-40c3-bf79-50ac2a7abb80-kube-api-access-5dq5x\") pod \"iptables-alerter-7b2vr\" (UID: \"b7bfcee0-0fd4-40c3-bf79-50ac2a7abb80\") " pod="openshift-network-operator/iptables-alerter-7b2vr" Sep 11 18:15:21.021496 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.021476 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g28hv\" (UniqueName: \"kubernetes.io/projected/a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d-kube-api-access-g28hv\") pod \"network-metrics-daemon-cmf6c\" (UID: \"a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d\") " pod="openshift-multus/network-metrics-daemon-cmf6c" Sep 11 18:15:21.021686 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.021667 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zp8v\" (UniqueName: \"kubernetes.io/projected/3c0fffdb-93cd-4936-96d6-1b4ef7d2486c-kube-api-access-9zp8v\") pod \"aws-ebs-csi-driver-node-fd24c\" (UID: \"3c0fffdb-93cd-4936-96d6-1b4ef7d2486c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fd24c" Sep 11 18:15:21.023678 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.023636 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkls2\" (UniqueName: \"kubernetes.io/projected/afbcd3d2-b5c0-474c-a177-e42b915d310d-kube-api-access-hkls2\") pod \"multus-76xpz\" (UID: \"afbcd3d2-b5c0-474c-a177-e42b915d310d\") " pod="openshift-multus/multus-76xpz" Sep 11 18:15:21.043156 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.043097 2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 18:10:20 +0000 UTC" deadline="2028-06-04 04:04:37.395824752 +0000 UTC" Sep 11 18:15:21.043156 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.043154 2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15153h49m16.352674881s" Sep 11 18:15:21.110477 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.110438 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4d1be3a0-f7d6-475b-b79f-c4fde3f4bcb9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m9j9m\" (UID: \"4d1be3a0-f7d6-475b-b79f-c4fde3f4bcb9\") " pod="openshift-multus/multus-additional-cni-plugins-m9j9m" Sep 11 18:15:21.110649 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.110491 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/42e5ddd8-a96c-45a1-adf9-caa47ee902db-run-openvswitch\") pod \"ovnkube-node-v4knd\" (UID: \"42e5ddd8-a96c-45a1-adf9-caa47ee902db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:21.110649 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.110524 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/72de70ec-e2ce-47b2-883d-75f4eb8eb39c-etc-sysctl-d\") pod \"tuned-nwpdb\" (UID: \"72de70ec-e2ce-47b2-883d-75f4eb8eb39c\") " pod="openshift-cluster-node-tuning-operator/tuned-nwpdb" Sep 11 18:15:21.110649 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.110548 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4d1be3a0-f7d6-475b-b79f-c4fde3f4bcb9-cnibin\") pod \"multus-additional-cni-plugins-m9j9m\" (UID: \"4d1be3a0-f7d6-475b-b79f-c4fde3f4bcb9\") " pod="openshift-multus/multus-additional-cni-plugins-m9j9m" Sep 11 18:15:21.110649 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.110572 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4d1be3a0-f7d6-475b-b79f-c4fde3f4bcb9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m9j9m\" (UID: \"4d1be3a0-f7d6-475b-b79f-c4fde3f4bcb9\") " pod="openshift-multus/multus-additional-cni-plugins-m9j9m" Sep 11 18:15:21.110649 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.110595 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/72de70ec-e2ce-47b2-883d-75f4eb8eb39c-etc-systemd\") pod \"tuned-nwpdb\" (UID: \"72de70ec-e2ce-47b2-883d-75f4eb8eb39c\") " pod="openshift-cluster-node-tuning-operator/tuned-nwpdb" Sep 11 18:15:21.110649 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.110619 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/42e5ddd8-a96c-45a1-adf9-caa47ee902db-log-socket\") pod \"ovnkube-node-v4knd\" (UID: \"42e5ddd8-a96c-45a1-adf9-caa47ee902db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:21.110649 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.110644 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/42e5ddd8-a96c-45a1-adf9-caa47ee902db-host-run-ovn-kubernetes\") pod \"ovnkube-node-v4knd\" (UID: \"42e5ddd8-a96c-45a1-adf9-caa47ee902db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:21.110984 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.110667 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/42e5ddd8-a96c-45a1-adf9-caa47ee902db-host-cni-bin\") pod \"ovnkube-node-v4knd\" (UID: \"42e5ddd8-a96c-45a1-adf9-caa47ee902db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:21.110984 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.110691 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/42e5ddd8-a96c-45a1-adf9-caa47ee902db-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v4knd\" (UID: \"42e5ddd8-a96c-45a1-adf9-caa47ee902db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:21.110984 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.110714 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/89de43e0-f41b-4570-af10-b3a3ef063480-kubelet-config\") pod \"global-pull-secret-syncer-sm2rw\" (UID: \"89de43e0-f41b-4570-af10-b3a3ef063480\") " pod="kube-system/global-pull-secret-syncer-sm2rw" Sep 11 18:15:21.110984 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.110738 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/42e5ddd8-a96c-45a1-adf9-caa47ee902db-var-lib-openvswitch\") pod \"ovnkube-node-v4knd\" (UID: \"42e5ddd8-a96c-45a1-adf9-caa47ee902db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:21.110984 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.110759 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/42e5ddd8-a96c-45a1-adf9-caa47ee902db-env-overrides\") pod \"ovnkube-node-v4knd\" (UID: \"42e5ddd8-a96c-45a1-adf9-caa47ee902db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:21.110984 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.110782 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/72de70ec-e2ce-47b2-883d-75f4eb8eb39c-host\") pod \"tuned-nwpdb\" (UID: \"72de70ec-e2ce-47b2-883d-75f4eb8eb39c\") " pod="openshift-cluster-node-tuning-operator/tuned-nwpdb" Sep 11 18:15:21.110984 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.110799 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4d1be3a0-f7d6-475b-b79f-c4fde3f4bcb9-cni-binary-copy\") pod \"multus-additional-cni-plugins-m9j9m\" (UID: \"4d1be3a0-f7d6-475b-b79f-c4fde3f4bcb9\") " pod="openshift-multus/multus-additional-cni-plugins-m9j9m" Sep 11 18:15:21.110984 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.110820 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/72de70ec-e2ce-47b2-883d-75f4eb8eb39c-sys\") pod \"tuned-nwpdb\" (UID: \"72de70ec-e2ce-47b2-883d-75f4eb8eb39c\") " pod="openshift-cluster-node-tuning-operator/tuned-nwpdb" Sep 11 18:15:21.110984 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.110839 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/72de70ec-e2ce-47b2-883d-75f4eb8eb39c-etc-tuned\") pod \"tuned-nwpdb\" (UID: \"72de70ec-e2ce-47b2-883d-75f4eb8eb39c\") " pod="openshift-cluster-node-tuning-operator/tuned-nwpdb" Sep 11 18:15:21.110984 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.110853 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/42e5ddd8-a96c-45a1-adf9-caa47ee902db-host-kubelet\") pod \"ovnkube-node-v4knd\" (UID: \"42e5ddd8-a96c-45a1-adf9-caa47ee902db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:21.110984 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.110902 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/42e5ddd8-a96c-45a1-adf9-caa47ee902db-run-systemd\") pod \"ovnkube-node-v4knd\" (UID: \"42e5ddd8-a96c-45a1-adf9-caa47ee902db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:21.110984 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.110918 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/42e5ddd8-a96c-45a1-adf9-caa47ee902db-ovnkube-script-lib\") pod \"ovnkube-node-v4knd\" (UID: \"42e5ddd8-a96c-45a1-adf9-caa47ee902db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:21.110984 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.110941 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/72de70ec-e2ce-47b2-883d-75f4eb8eb39c-lib-modules\") pod \"tuned-nwpdb\" (UID: \"72de70ec-e2ce-47b2-883d-75f4eb8eb39c\") " pod="openshift-cluster-node-tuning-operator/tuned-nwpdb" Sep 11 18:15:21.110984 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.110960 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/42e5ddd8-a96c-45a1-adf9-caa47ee902db-ovnkube-config\") pod \"ovnkube-node-v4knd\" (UID: \"42e5ddd8-a96c-45a1-adf9-caa47ee902db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:21.110984 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.110976 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/42e5ddd8-a96c-45a1-adf9-caa47ee902db-ovn-node-metrics-cert\") pod \"ovnkube-node-v4knd\" (UID: \"42e5ddd8-a96c-45a1-adf9-caa47ee902db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:21.110984 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.110990 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/89de43e0-f41b-4570-af10-b3a3ef063480-dbus\") pod \"global-pull-secret-syncer-sm2rw\" (UID: \"89de43e0-f41b-4570-af10-b3a3ef063480\") " pod="kube-system/global-pull-secret-syncer-sm2rw" Sep 11 18:15:21.111805 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.111013 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/72de70ec-e2ce-47b2-883d-75f4eb8eb39c-run\") pod \"tuned-nwpdb\" (UID: \"72de70ec-e2ce-47b2-883d-75f4eb8eb39c\") " pod="openshift-cluster-node-tuning-operator/tuned-nwpdb" Sep 11 18:15:21.111805 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.111033 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl2fr\" (UniqueName: \"kubernetes.io/projected/4d1be3a0-f7d6-475b-b79f-c4fde3f4bcb9-kube-api-access-sl2fr\") pod \"multus-additional-cni-plugins-m9j9m\" (UID: \"4d1be3a0-f7d6-475b-b79f-c4fde3f4bcb9\") " pod="openshift-multus/multus-additional-cni-plugins-m9j9m" Sep 11 18:15:21.111805 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.111064 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/72de70ec-e2ce-47b2-883d-75f4eb8eb39c-etc-modprobe-d\") pod \"tuned-nwpdb\" (UID: \"72de70ec-e2ce-47b2-883d-75f4eb8eb39c\") " pod="openshift-cluster-node-tuning-operator/tuned-nwpdb" Sep 11 18:15:21.111805 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.111087 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj82v\" (UniqueName: \"kubernetes.io/projected/72de70ec-e2ce-47b2-883d-75f4eb8eb39c-kube-api-access-sj82v\") pod \"tuned-nwpdb\" (UID: \"72de70ec-e2ce-47b2-883d-75f4eb8eb39c\") " pod="openshift-cluster-node-tuning-operator/tuned-nwpdb" Sep 11 18:15:21.111805 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.111132 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d1be3a0-f7d6-475b-b79f-c4fde3f4bcb9-system-cni-dir\") pod \"multus-additional-cni-plugins-m9j9m\" (UID: \"4d1be3a0-f7d6-475b-b79f-c4fde3f4bcb9\") " pod="openshift-multus/multus-additional-cni-plugins-m9j9m" Sep 11 18:15:21.111805 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.111183 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/72de70ec-e2ce-47b2-883d-75f4eb8eb39c-host\") pod \"tuned-nwpdb\" (UID: \"72de70ec-e2ce-47b2-883d-75f4eb8eb39c\") " pod="openshift-cluster-node-tuning-operator/tuned-nwpdb" Sep 11 18:15:21.111805 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.111226 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/72de70ec-e2ce-47b2-883d-75f4eb8eb39c-run\") pod \"tuned-nwpdb\" (UID: \"72de70ec-e2ce-47b2-883d-75f4eb8eb39c\") " pod="openshift-cluster-node-tuning-operator/tuned-nwpdb" Sep 11 18:15:21.111805 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.111284 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72de70ec-e2ce-47b2-883d-75f4eb8eb39c-etc-kubernetes\") pod \"tuned-nwpdb\" (UID: \"72de70ec-e2ce-47b2-883d-75f4eb8eb39c\") " pod="openshift-cluster-node-tuning-operator/tuned-nwpdb" Sep 11 18:15:21.111805 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.111378 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/42e5ddd8-a96c-45a1-adf9-caa47ee902db-host-kubelet\") pod \"ovnkube-node-v4knd\" (UID: \"42e5ddd8-a96c-45a1-adf9-caa47ee902db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:21.111805 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.111417 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/42e5ddd8-a96c-45a1-adf9-caa47ee902db-run-systemd\") pod \"ovnkube-node-v4knd\" (UID: \"42e5ddd8-a96c-45a1-adf9-caa47ee902db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:21.111805 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.111535 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/72de70ec-e2ce-47b2-883d-75f4eb8eb39c-sys\") pod \"tuned-nwpdb\" (UID: \"72de70ec-e2ce-47b2-883d-75f4eb8eb39c\") " pod="openshift-cluster-node-tuning-operator/tuned-nwpdb" Sep 11 18:15:21.111805 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.111563 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/42e5ddd8-a96c-45a1-adf9-caa47ee902db-log-socket\") pod \"ovnkube-node-v4knd\" (UID: \"42e5ddd8-a96c-45a1-adf9-caa47ee902db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:21.111805 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.111609 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/42e5ddd8-a96c-45a1-adf9-caa47ee902db-run-openvswitch\") pod \"ovnkube-node-v4knd\" (UID: \"42e5ddd8-a96c-45a1-adf9-caa47ee902db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:21.111805 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.111722 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/42e5ddd8-a96c-45a1-adf9-caa47ee902db-ovnkube-config\") pod \"ovnkube-node-v4knd\" (UID: \"42e5ddd8-a96c-45a1-adf9-caa47ee902db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:21.111805 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.111740 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4d1be3a0-f7d6-475b-b79f-c4fde3f4bcb9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m9j9m\" (UID: \"4d1be3a0-f7d6-475b-b79f-c4fde3f4bcb9\") " pod="openshift-multus/multus-additional-cni-plugins-m9j9m" Sep 11 18:15:21.111805 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.111763 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/72de70ec-e2ce-47b2-883d-75f4eb8eb39c-etc-sysctl-d\") pod \"tuned-nwpdb\" (UID: \"72de70ec-e2ce-47b2-883d-75f4eb8eb39c\") " pod="openshift-cluster-node-tuning-operator/tuned-nwpdb" Sep 11 18:15:21.111805 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.111790 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/42e5ddd8-a96c-45a1-adf9-caa47ee902db-host-run-ovn-kubernetes\") pod \"ovnkube-node-v4knd\" (UID: \"42e5ddd8-a96c-45a1-adf9-caa47ee902db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:21.112645 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.111825 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/72de70ec-e2ce-47b2-883d-75f4eb8eb39c-etc-modprobe-d\") pod \"tuned-nwpdb\" (UID: \"72de70ec-e2ce-47b2-883d-75f4eb8eb39c\") " pod="openshift-cluster-node-tuning-operator/tuned-nwpdb" Sep 11 18:15:21.112645 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.111845 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/89de43e0-f41b-4570-af10-b3a3ef063480-kubelet-config\") pod \"global-pull-secret-syncer-sm2rw\" (UID: \"89de43e0-f41b-4570-af10-b3a3ef063480\") " pod="kube-system/global-pull-secret-syncer-sm2rw" Sep 11 18:15:21.112645 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.111865 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/42e5ddd8-a96c-45a1-adf9-caa47ee902db-systemd-units\") pod \"ovnkube-node-v4knd\" (UID: \"42e5ddd8-a96c-45a1-adf9-caa47ee902db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:21.112645 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.111909 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/42e5ddd8-a96c-45a1-adf9-caa47ee902db-systemd-units\") pod \"ovnkube-node-v4knd\" (UID: \"42e5ddd8-a96c-45a1-adf9-caa47ee902db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:21.112645 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.111943 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/42e5ddd8-a96c-45a1-adf9-caa47ee902db-host-cni-bin\") pod \"ovnkube-node-v4knd\" (UID: \"42e5ddd8-a96c-45a1-adf9-caa47ee902db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:21.112645 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.111957 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/72de70ec-e2ce-47b2-883d-75f4eb8eb39c-lib-modules\") pod \"tuned-nwpdb\" (UID: \"72de70ec-e2ce-47b2-883d-75f4eb8eb39c\") " pod="openshift-cluster-node-tuning-operator/tuned-nwpdb" Sep 11 18:15:21.112645 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.111973 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4d1be3a0-f7d6-475b-b79f-c4fde3f4bcb9-cnibin\") pod \"multus-additional-cni-plugins-m9j9m\" (UID: \"4d1be3a0-f7d6-475b-b79f-c4fde3f4bcb9\") " pod="openshift-multus/multus-additional-cni-plugins-m9j9m" Sep 11 18:15:21.112645 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.111989 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/42e5ddd8-a96c-45a1-adf9-caa47ee902db-node-log\") pod \"ovnkube-node-v4knd\" (UID: \"42e5ddd8-a96c-45a1-adf9-caa47ee902db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:21.112645 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.111979 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4d1be3a0-f7d6-475b-b79f-c4fde3f4bcb9-cni-binary-copy\") pod \"multus-additional-cni-plugins-m9j9m\" (UID: \"4d1be3a0-f7d6-475b-b79f-c4fde3f4bcb9\") " pod="openshift-multus/multus-additional-cni-plugins-m9j9m" Sep 11 18:15:21.112645 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.112068 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/42e5ddd8-a96c-45a1-adf9-caa47ee902db-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v4knd\" (UID: \"42e5ddd8-a96c-45a1-adf9-caa47ee902db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:21.112645 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.112131 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d1be3a0-f7d6-475b-b79f-c4fde3f4bcb9-system-cni-dir\") pod \"multus-additional-cni-plugins-m9j9m\" (UID: \"4d1be3a0-f7d6-475b-b79f-c4fde3f4bcb9\") " pod="openshift-multus/multus-additional-cni-plugins-m9j9m" Sep 11 18:15:21.112645 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.112135 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4d1be3a0-f7d6-475b-b79f-c4fde3f4bcb9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m9j9m\" (UID: \"4d1be3a0-f7d6-475b-b79f-c4fde3f4bcb9\") " pod="openshift-multus/multus-additional-cni-plugins-m9j9m" Sep 11 18:15:21.112645 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.112151 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/42e5ddd8-a96c-45a1-adf9-caa47ee902db-env-overrides\") pod \"ovnkube-node-v4knd\" (UID: \"42e5ddd8-a96c-45a1-adf9-caa47ee902db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:21.112645 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.112168 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/42e5ddd8-a96c-45a1-adf9-caa47ee902db-host-slash\") pod \"ovnkube-node-v4knd\" (UID: \"42e5ddd8-a96c-45a1-adf9-caa47ee902db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:21.112645 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.112170 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/89de43e0-f41b-4570-af10-b3a3ef063480-dbus\") pod \"global-pull-secret-syncer-sm2rw\" (UID: \"89de43e0-f41b-4570-af10-b3a3ef063480\") " pod="kube-system/global-pull-secret-syncer-sm2rw" Sep 11 18:15:21.112645 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.112179 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/42e5ddd8-a96c-45a1-adf9-caa47ee902db-node-log\") pod \"ovnkube-node-v4knd\" (UID: \"42e5ddd8-a96c-45a1-adf9-caa47ee902db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:21.112645 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.112208 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/42e5ddd8-a96c-45a1-adf9-caa47ee902db-host-slash\") pod \"ovnkube-node-v4knd\" (UID: \"42e5ddd8-a96c-45a1-adf9-caa47ee902db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:21.113454 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.112225 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/72de70ec-e2ce-47b2-883d-75f4eb8eb39c-etc-systemd\") pod \"tuned-nwpdb\" (UID: \"72de70ec-e2ce-47b2-883d-75f4eb8eb39c\") " pod="openshift-cluster-node-tuning-operator/tuned-nwpdb" Sep 11 18:15:21.113454 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.112229 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72de70ec-e2ce-47b2-883d-75f4eb8eb39c-etc-kubernetes\") pod \"tuned-nwpdb\" (UID: \"72de70ec-e2ce-47b2-883d-75f4eb8eb39c\") " pod="openshift-cluster-node-tuning-operator/tuned-nwpdb" Sep 11 18:15:21.113454 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.112231 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/42e5ddd8-a96c-45a1-adf9-caa47ee902db-host-run-netns\") pod \"ovnkube-node-v4knd\" (UID: \"42e5ddd8-a96c-45a1-adf9-caa47ee902db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:21.113454 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.112261 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/42e5ddd8-a96c-45a1-adf9-caa47ee902db-host-run-netns\") pod \"ovnkube-node-v4knd\" (UID: \"42e5ddd8-a96c-45a1-adf9-caa47ee902db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:21.113454 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.112272 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/42e5ddd8-a96c-45a1-adf9-caa47ee902db-run-ovn\") pod \"ovnkube-node-v4knd\" (UID: \"42e5ddd8-a96c-45a1-adf9-caa47ee902db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:21.113454 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.112275 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/42e5ddd8-a96c-45a1-adf9-caa47ee902db-var-lib-openvswitch\") pod \"ovnkube-node-v4knd\" (UID: \"42e5ddd8-a96c-45a1-adf9-caa47ee902db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:21.113454 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.112303 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/42e5ddd8-a96c-45a1-adf9-caa47ee902db-run-ovn\") pod \"ovnkube-node-v4knd\" (UID: \"42e5ddd8-a96c-45a1-adf9-caa47ee902db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:21.113454 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.112334 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/72de70ec-e2ce-47b2-883d-75f4eb8eb39c-tmp\") pod \"tuned-nwpdb\" (UID: \"72de70ec-e2ce-47b2-883d-75f4eb8eb39c\") " pod="openshift-cluster-node-tuning-operator/tuned-nwpdb" Sep 11 18:15:21.113454 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.112359 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/42e5ddd8-a96c-45a1-adf9-caa47ee902db-etc-openvswitch\") pod \"ovnkube-node-v4knd\" (UID: \"42e5ddd8-a96c-45a1-adf9-caa47ee902db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:21.113454 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.112360 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/42e5ddd8-a96c-45a1-adf9-caa47ee902db-ovnkube-script-lib\") pod \"ovnkube-node-v4knd\" (UID: \"42e5ddd8-a96c-45a1-adf9-caa47ee902db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:21.113454 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.112392 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnwww\" (UniqueName: \"kubernetes.io/projected/42e5ddd8-a96c-45a1-adf9-caa47ee902db-kube-api-access-cnwww\") pod \"ovnkube-node-v4knd\" (UID: \"42e5ddd8-a96c-45a1-adf9-caa47ee902db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:21.113454 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.112413 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/42e5ddd8-a96c-45a1-adf9-caa47ee902db-etc-openvswitch\") pod \"ovnkube-node-v4knd\" (UID: \"42e5ddd8-a96c-45a1-adf9-caa47ee902db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:21.113454 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.112457 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/89de43e0-f41b-4570-af10-b3a3ef063480-original-pull-secret\") pod \"global-pull-secret-syncer-sm2rw\" (UID: \"89de43e0-f41b-4570-af10-b3a3ef063480\") " pod="kube-system/global-pull-secret-syncer-sm2rw" Sep 11 18:15:21.113454 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:21.112626 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 18:15:21.113454 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.112639 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/72de70ec-e2ce-47b2-883d-75f4eb8eb39c-etc-sysconfig\") pod \"tuned-nwpdb\" (UID: \"72de70ec-e2ce-47b2-883d-75f4eb8eb39c\") " pod="openshift-cluster-node-tuning-operator/tuned-nwpdb" Sep 11 18:15:21.113454 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:21.112692 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89de43e0-f41b-4570-af10-b3a3ef063480-original-pull-secret podName:89de43e0-f41b-4570-af10-b3a3ef063480 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:21.612675134 +0000 UTC m=+3.109692881 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/89de43e0-f41b-4570-af10-b3a3ef063480-original-pull-secret") pod "global-pull-secret-syncer-sm2rw" (UID: "89de43e0-f41b-4570-af10-b3a3ef063480") : object "kube-system"/"original-pull-secret" not registered Sep 11 18:15:21.113454 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.112752 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/72de70ec-e2ce-47b2-883d-75f4eb8eb39c-etc-sysctl-conf\") pod \"tuned-nwpdb\" (UID: \"72de70ec-e2ce-47b2-883d-75f4eb8eb39c\") " pod="openshift-cluster-node-tuning-operator/tuned-nwpdb" Sep 11 18:15:21.114279 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.112767 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/72de70ec-e2ce-47b2-883d-75f4eb8eb39c-etc-sysconfig\") pod \"tuned-nwpdb\" (UID: \"72de70ec-e2ce-47b2-883d-75f4eb8eb39c\") " pod="openshift-cluster-node-tuning-operator/tuned-nwpdb" Sep 11 18:15:21.114279 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.112806 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/72de70ec-e2ce-47b2-883d-75f4eb8eb39c-var-lib-kubelet\") pod \"tuned-nwpdb\" (UID: \"72de70ec-e2ce-47b2-883d-75f4eb8eb39c\") " pod="openshift-cluster-node-tuning-operator/tuned-nwpdb" Sep 11 18:15:21.114279 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.112832 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4d1be3a0-f7d6-475b-b79f-c4fde3f4bcb9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m9j9m\" (UID: \"4d1be3a0-f7d6-475b-b79f-c4fde3f4bcb9\") " pod="openshift-multus/multus-additional-cni-plugins-m9j9m" Sep 11 18:15:21.114279 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.112877 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/42e5ddd8-a96c-45a1-adf9-caa47ee902db-host-cni-netd\") pod \"ovnkube-node-v4knd\" (UID: \"42e5ddd8-a96c-45a1-adf9-caa47ee902db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:21.114279 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.112947 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/42e5ddd8-a96c-45a1-adf9-caa47ee902db-host-cni-netd\") pod \"ovnkube-node-v4knd\" (UID: \"42e5ddd8-a96c-45a1-adf9-caa47ee902db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:21.114279 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.112962 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/72de70ec-e2ce-47b2-883d-75f4eb8eb39c-etc-sysctl-conf\") pod \"tuned-nwpdb\" (UID: \"72de70ec-e2ce-47b2-883d-75f4eb8eb39c\") " pod="openshift-cluster-node-tuning-operator/tuned-nwpdb" Sep 11 18:15:21.114279 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.112995 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/72de70ec-e2ce-47b2-883d-75f4eb8eb39c-var-lib-kubelet\") pod \"tuned-nwpdb\" (UID: \"72de70ec-e2ce-47b2-883d-75f4eb8eb39c\") " pod="openshift-cluster-node-tuning-operator/tuned-nwpdb" Sep 11 18:15:21.114279 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.113262 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4d1be3a0-f7d6-475b-b79f-c4fde3f4bcb9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m9j9m\" (UID: \"4d1be3a0-f7d6-475b-b79f-c4fde3f4bcb9\") " pod="openshift-multus/multus-additional-cni-plugins-m9j9m" Sep 11 18:15:21.116089 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.115838 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/72de70ec-e2ce-47b2-883d-75f4eb8eb39c-etc-tuned\") pod \"tuned-nwpdb\" (UID: \"72de70ec-e2ce-47b2-883d-75f4eb8eb39c\") " pod="openshift-cluster-node-tuning-operator/tuned-nwpdb" Sep 11 18:15:21.116089 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.116077 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/42e5ddd8-a96c-45a1-adf9-caa47ee902db-ovn-node-metrics-cert\") pod \"ovnkube-node-v4knd\" (UID: \"42e5ddd8-a96c-45a1-adf9-caa47ee902db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:21.117038 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.116966 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/72de70ec-e2ce-47b2-883d-75f4eb8eb39c-tmp\") pod \"tuned-nwpdb\" (UID: \"72de70ec-e2ce-47b2-883d-75f4eb8eb39c\") " pod="openshift-cluster-node-tuning-operator/tuned-nwpdb" Sep 11 18:15:21.124838 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.124808 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl2fr\" (UniqueName: \"kubernetes.io/projected/4d1be3a0-f7d6-475b-b79f-c4fde3f4bcb9-kube-api-access-sl2fr\") pod \"multus-additional-cni-plugins-m9j9m\" (UID: \"4d1be3a0-f7d6-475b-b79f-c4fde3f4bcb9\") " pod="openshift-multus/multus-additional-cni-plugins-m9j9m" Sep 11 18:15:21.125485 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.125461 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj82v\" (UniqueName: \"kubernetes.io/projected/72de70ec-e2ce-47b2-883d-75f4eb8eb39c-kube-api-access-sj82v\") pod \"tuned-nwpdb\" (UID: \"72de70ec-e2ce-47b2-883d-75f4eb8eb39c\") " pod="openshift-cluster-node-tuning-operator/tuned-nwpdb" Sep 11 18:15:21.125934 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.125896 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnwww\" (UniqueName: \"kubernetes.io/projected/42e5ddd8-a96c-45a1-adf9-caa47ee902db-kube-api-access-cnwww\") pod \"ovnkube-node-v4knd\" (UID: \"42e5ddd8-a96c-45a1-adf9-caa47ee902db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:21.137501 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.137450 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-128.ec2.internal" event={"ID":"ebf313d069610fccaa30accef842e07e","Type":"ContainerStarted","Data":"1ad65994cbaea40d1e21e70ac2402a41aca7e525e2d48afb372b870be6b280a1"} Sep 11 18:15:21.139083 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.139039 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-128.ec2.internal" event={"ID":"577bf0a7d48f3dd63acd7590896a721f","Type":"ContainerStarted","Data":"e9ce59465e0215deac3ebd4d435129da65375f3671be53f66ec3980fefc49ef8"} Sep 11 18:15:21.188913 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.188784 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-76xpz" Sep 11 18:15:21.198167 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.198137 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7b2vr" Sep 11 18:15:21.203671 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:21.203640 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafbcd3d2_b5c0_474c_a177_e42b915d310d.slice/crio-874298710a04c33339a11943b81593217b1df366fb93cb2237cbf1de0d0f697b WatchSource:0}: Error finding container 874298710a04c33339a11943b81593217b1df366fb93cb2237cbf1de0d0f697b: Status 404 returned error can't find the container with id 874298710a04c33339a11943b81593217b1df366fb93cb2237cbf1de0d0f697b Sep 11 18:15:21.207490 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.207469 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gt5ln" Sep 11 18:15:21.213939 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:21.213913 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7bfcee0_0fd4_40c3_bf79_50ac2a7abb80.slice/crio-a6b054251e8242d7ab9b2f8741b3660e096a724c3bfc7b13238459c7010ebd43 WatchSource:0}: Error finding container a6b054251e8242d7ab9b2f8741b3660e096a724c3bfc7b13238459c7010ebd43: Status 404 returned error can't find the container with id a6b054251e8242d7ab9b2f8741b3660e096a724c3bfc7b13238459c7010ebd43 Sep 11 18:15:21.214889 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.214865 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:21.220663 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.220421 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fd24c" Sep 11 18:15:21.221654 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:21.221475 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9c4d02b_cc0d_4812_8fe2_80f5276923ba.slice/crio-b76546303121172ac6795ed592a0689e14ab22753c6e93f1779c69b2088d196b WatchSource:0}: Error finding container b76546303121172ac6795ed592a0689e14ab22753c6e93f1779c69b2088d196b: Status 404 returned error can't find the container with id b76546303121172ac6795ed592a0689e14ab22753c6e93f1779c69b2088d196b Sep 11 18:15:21.226622 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:21.226596 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42e5ddd8_a96c_45a1_adf9_caa47ee902db.slice/crio-22bde6db42e35bd006f3daddeff63f7109eaee558675a85e81a7b0bf6b16d302 WatchSource:0}: Error finding container 22bde6db42e35bd006f3daddeff63f7109eaee558675a85e81a7b0bf6b16d302: Status 404 returned error can't find the container with id 22bde6db42e35bd006f3daddeff63f7109eaee558675a85e81a7b0bf6b16d302 Sep 11 18:15:21.227755 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.227734 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xh28n" Sep 11 18:15:21.234601 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.234474 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-k7qjg" Sep 11 18:15:21.237752 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:21.237720 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c0fffdb_93cd_4936_96d6_1b4ef7d2486c.slice/crio-efc3c312bc7c63246c2deb215b7b745aa4b35a8243ce1ed9cce46d5a897b10ea WatchSource:0}: Error finding container efc3c312bc7c63246c2deb215b7b745aa4b35a8243ce1ed9cce46d5a897b10ea: Status 404 returned error can't find the container with id efc3c312bc7c63246c2deb215b7b745aa4b35a8243ce1ed9cce46d5a897b10ea Sep 11 18:15:21.241867 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.241849 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-m9j9m" Sep 11 18:15:21.247579 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.247431 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nwpdb" Sep 11 18:15:21.250088 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:21.250013 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6547cc6d_cb31_474a_8065_7fd47766fe02.slice/crio-93027775220b31f7ecca73a0efec20616f85c37f971dce3eaa4e2376d41ed5dd WatchSource:0}: Error finding container 93027775220b31f7ecca73a0efec20616f85c37f971dce3eaa4e2376d41ed5dd: Status 404 returned error can't find the container with id 93027775220b31f7ecca73a0efec20616f85c37f971dce3eaa4e2376d41ed5dd Sep 11 18:15:21.516380 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.516292 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d-metrics-certs\") pod \"network-metrics-daemon-cmf6c\" (UID: \"a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d\") " pod="openshift-multus/network-metrics-daemon-cmf6c" Sep 11 18:15:21.516380 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.516358 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g69tq\" (UniqueName: \"kubernetes.io/projected/e183b6fd-207a-44ed-829a-6872d65d0a57-kube-api-access-g69tq\") pod \"network-check-target-zr7xc\" (UID: \"e183b6fd-207a-44ed-829a-6872d65d0a57\") " pod="openshift-network-diagnostics/network-check-target-zr7xc" Sep 11 18:15:21.516602 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:21.516429 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 18:15:21.516602 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:21.516516 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d-metrics-certs podName:a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d nodeName:}" failed. No retries permitted until 2026-09-11 18:15:22.516496347 +0000 UTC m=+4.013514079 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d-metrics-certs") pod "network-metrics-daemon-cmf6c" (UID: "a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 18:15:21.516602 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:21.516534 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 18:15:21.516602 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:21.516561 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 18:15:21.516602 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:21.516575 2561 projected.go:196] Error preparing data for projected volume kube-api-access-g69tq for pod openshift-network-diagnostics/network-check-target-zr7xc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 18:15:21.516839 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:21.516631 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e183b6fd-207a-44ed-829a-6872d65d0a57-kube-api-access-g69tq podName:e183b6fd-207a-44ed-829a-6872d65d0a57 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:22.516610817 +0000 UTC m=+4.013628628 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-g69tq" (UniqueName: "kubernetes.io/projected/e183b6fd-207a-44ed-829a-6872d65d0a57-kube-api-access-g69tq") pod "network-check-target-zr7xc" (UID: "e183b6fd-207a-44ed-829a-6872d65d0a57") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 18:15:21.617268 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:21.617230 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/89de43e0-f41b-4570-af10-b3a3ef063480-original-pull-secret\") pod \"global-pull-secret-syncer-sm2rw\" (UID: \"89de43e0-f41b-4570-af10-b3a3ef063480\") " pod="kube-system/global-pull-secret-syncer-sm2rw" Sep 11 18:15:21.617459 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:21.617428 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 18:15:21.617530 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:21.617518 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89de43e0-f41b-4570-af10-b3a3ef063480-original-pull-secret podName:89de43e0-f41b-4570-af10-b3a3ef063480 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:22.617497462 +0000 UTC m=+4.114515215 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/89de43e0-f41b-4570-af10-b3a3ef063480-original-pull-secret") pod "global-pull-secret-syncer-sm2rw" (UID: "89de43e0-f41b-4570-af10-b3a3ef063480") : object "kube-system"/"original-pull-secret" not registered Sep 11 18:15:22.044378 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:22.044292 2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 18:10:20 +0000 UTC" deadline="2028-02-15 02:56:38.16584093 +0000 UTC" Sep 11 18:15:22.044378 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:22.044356 2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12512h41m16.121489574s" Sep 11 18:15:22.143337 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:22.143261 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-k7qjg" event={"ID":"6547cc6d-cb31-474a-8065-7fd47766fe02","Type":"ContainerStarted","Data":"93027775220b31f7ecca73a0efec20616f85c37f971dce3eaa4e2376d41ed5dd"} Sep 11 18:15:22.145863 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:22.145831 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-76xpz" event={"ID":"afbcd3d2-b5c0-474c-a177-e42b915d310d","Type":"ContainerStarted","Data":"874298710a04c33339a11943b81593217b1df366fb93cb2237cbf1de0d0f697b"} Sep 11 18:15:22.147666 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:22.147637 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xh28n" event={"ID":"7f7d5659-22fb-43f1-8970-47837d126795","Type":"ContainerStarted","Data":"d55717ef3520b7a9e8c64fcd80058c458621e39a322fcba162792b43f3aa032b"} Sep 11 18:15:22.149249 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:22.149221 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fd24c" event={"ID":"3c0fffdb-93cd-4936-96d6-1b4ef7d2486c","Type":"ContainerStarted","Data":"efc3c312bc7c63246c2deb215b7b745aa4b35a8243ce1ed9cce46d5a897b10ea"} Sep 11 18:15:22.150855 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:22.150827 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" event={"ID":"42e5ddd8-a96c-45a1-adf9-caa47ee902db","Type":"ContainerStarted","Data":"22bde6db42e35bd006f3daddeff63f7109eaee558675a85e81a7b0bf6b16d302"} Sep 11 18:15:22.152486 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:22.152417 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gt5ln" event={"ID":"e9c4d02b-cc0d-4812-8fe2-80f5276923ba","Type":"ContainerStarted","Data":"b76546303121172ac6795ed592a0689e14ab22753c6e93f1779c69b2088d196b"} Sep 11 18:15:22.154425 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:22.154378 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7b2vr" event={"ID":"b7bfcee0-0fd4-40c3-bf79-50ac2a7abb80","Type":"ContainerStarted","Data":"a6b054251e8242d7ab9b2f8741b3660e096a724c3bfc7b13238459c7010ebd43"} Sep 11 18:15:22.524492 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:22.524212 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d-metrics-certs\") pod \"network-metrics-daemon-cmf6c\" (UID: \"a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d\") " pod="openshift-multus/network-metrics-daemon-cmf6c" Sep 11 18:15:22.524685 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:22.524542 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g69tq\" (UniqueName: \"kubernetes.io/projected/e183b6fd-207a-44ed-829a-6872d65d0a57-kube-api-access-g69tq\") pod \"network-check-target-zr7xc\" (UID: \"e183b6fd-207a-44ed-829a-6872d65d0a57\") " pod="openshift-network-diagnostics/network-check-target-zr7xc" Sep 11 18:15:22.524755 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:22.524410 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 18:15:22.524808 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:22.524760 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d-metrics-certs podName:a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d nodeName:}" failed. No retries permitted until 2026-09-11 18:15:24.524741269 +0000 UTC m=+6.021759004 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d-metrics-certs") pod "network-metrics-daemon-cmf6c" (UID: "a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 18:15:22.525201 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:22.524693 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 18:15:22.525295 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:22.525218 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 18:15:22.525295 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:22.525235 2561 projected.go:196] Error preparing data for projected volume kube-api-access-g69tq for pod openshift-network-diagnostics/network-check-target-zr7xc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 18:15:22.525295 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:22.525285 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e183b6fd-207a-44ed-829a-6872d65d0a57-kube-api-access-g69tq podName:e183b6fd-207a-44ed-829a-6872d65d0a57 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:24.525268957 +0000 UTC m=+6.022286701 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-g69tq" (UniqueName: "kubernetes.io/projected/e183b6fd-207a-44ed-829a-6872d65d0a57-kube-api-access-g69tq") pod "network-check-target-zr7xc" (UID: "e183b6fd-207a-44ed-829a-6872d65d0a57") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 18:15:22.573922 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:22.573889 2561 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 18:15:22.598203 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:22.595599 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d1be3a0_f7d6_475b_b79f_c4fde3f4bcb9.slice/crio-bce7ff98839447187beb4dab15e2b9eb6284430d6126f807e9c4d171091289f4 WatchSource:0}: Error finding container bce7ff98839447187beb4dab15e2b9eb6284430d6126f807e9c4d171091289f4: Status 404 returned error can't find the container with id bce7ff98839447187beb4dab15e2b9eb6284430d6126f807e9c4d171091289f4 Sep 11 18:15:22.613653 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:22.613614 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72de70ec_e2ce_47b2_883d_75f4eb8eb39c.slice/crio-d412e5d2743ed30756be5861f2e0e6b3308451fe2eec96f79decf2decd2b7a5c WatchSource:0}: Error finding container d412e5d2743ed30756be5861f2e0e6b3308451fe2eec96f79decf2decd2b7a5c: Status 404 returned error can't find the container with id d412e5d2743ed30756be5861f2e0e6b3308451fe2eec96f79decf2decd2b7a5c Sep 11 18:15:22.625489 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:22.624943 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/89de43e0-f41b-4570-af10-b3a3ef063480-original-pull-secret\") pod \"global-pull-secret-syncer-sm2rw\" (UID: \"89de43e0-f41b-4570-af10-b3a3ef063480\") " pod="kube-system/global-pull-secret-syncer-sm2rw" Sep 11 18:15:22.625489 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:22.625123 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 18:15:22.625489 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:22.625188 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89de43e0-f41b-4570-af10-b3a3ef063480-original-pull-secret podName:89de43e0-f41b-4570-af10-b3a3ef063480 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:24.625167733 +0000 UTC m=+6.122185471 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/89de43e0-f41b-4570-af10-b3a3ef063480-original-pull-secret") pod "global-pull-secret-syncer-sm2rw" (UID: "89de43e0-f41b-4570-af10-b3a3ef063480") : object "kube-system"/"original-pull-secret" not registered Sep 11 18:15:23.132894 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:23.132818 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr7xc" Sep 11 18:15:23.133561 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:23.132949 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr7xc" podUID="e183b6fd-207a-44ed-829a-6872d65d0a57" Sep 11 18:15:23.133561 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:23.133390 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cmf6c" Sep 11 18:15:23.133561 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:23.133497 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cmf6c" podUID="a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d" Sep 11 18:15:23.133723 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:23.133565 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm2rw" Sep 11 18:15:23.133723 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:23.133639 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm2rw" podUID="89de43e0-f41b-4570-af10-b3a3ef063480" Sep 11 18:15:23.174872 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:23.174826 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m9j9m" event={"ID":"4d1be3a0-f7d6-475b-b79f-c4fde3f4bcb9","Type":"ContainerStarted","Data":"bce7ff98839447187beb4dab15e2b9eb6284430d6126f807e9c4d171091289f4"} Sep 11 18:15:23.184137 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:23.183665 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-128.ec2.internal" event={"ID":"577bf0a7d48f3dd63acd7590896a721f","Type":"ContainerStarted","Data":"c662453574a253b6b9fd7581e1ef7c64915d4741f4b4c9747078a3a20b00bc25"} Sep 11 18:15:23.185277 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:23.185242 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nwpdb" event={"ID":"72de70ec-e2ce-47b2-883d-75f4eb8eb39c","Type":"ContainerStarted","Data":"d412e5d2743ed30756be5861f2e0e6b3308451fe2eec96f79decf2decd2b7a5c"} Sep 11 18:15:23.200084 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:23.199627 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-128.ec2.internal" podStartSLOduration=4.19960789 podStartE2EDuration="4.19960789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:23.199448528 +0000 UTC m=+4.696466283" watchObservedRunningTime="2026-09-11 18:15:23.19960789 +0000 UTC m=+4.696625646" Sep 11 18:15:24.195719 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:24.194632 2561 generic.go:356] "Generic (PLEG): container finished" podID="ebf313d069610fccaa30accef842e07e" containerID="862e9ecfd5bed2eb67c8c4a8f6b51ea9b1bf6e76eedc753eb86184a54504c932" exitCode=0 Sep 11 18:15:24.195719 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:24.195288 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-128.ec2.internal" event={"ID":"ebf313d069610fccaa30accef842e07e","Type":"ContainerDied","Data":"862e9ecfd5bed2eb67c8c4a8f6b51ea9b1bf6e76eedc753eb86184a54504c932"} Sep 11 18:15:24.542303 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:24.542210 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d-metrics-certs\") pod \"network-metrics-daemon-cmf6c\" (UID: \"a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d\") " pod="openshift-multus/network-metrics-daemon-cmf6c" Sep 11 18:15:24.542303 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:24.542286 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g69tq\" (UniqueName: \"kubernetes.io/projected/e183b6fd-207a-44ed-829a-6872d65d0a57-kube-api-access-g69tq\") pod \"network-check-target-zr7xc\" (UID: \"e183b6fd-207a-44ed-829a-6872d65d0a57\") " pod="openshift-network-diagnostics/network-check-target-zr7xc" Sep 11 18:15:24.542530 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:24.542460 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 18:15:24.542530 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:24.542481 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 18:15:24.542530 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:24.542494 2561 projected.go:196] Error preparing data for projected volume kube-api-access-g69tq for pod openshift-network-diagnostics/network-check-target-zr7xc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 18:15:24.542682 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:24.542551 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e183b6fd-207a-44ed-829a-6872d65d0a57-kube-api-access-g69tq podName:e183b6fd-207a-44ed-829a-6872d65d0a57 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:28.542533469 +0000 UTC m=+10.039551224 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-g69tq" (UniqueName: "kubernetes.io/projected/e183b6fd-207a-44ed-829a-6872d65d0a57-kube-api-access-g69tq") pod "network-check-target-zr7xc" (UID: "e183b6fd-207a-44ed-829a-6872d65d0a57") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 18:15:24.542981 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:24.542958 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 18:15:24.543102 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:24.543016 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d-metrics-certs podName:a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d nodeName:}" failed. No retries permitted until 2026-09-11 18:15:28.543001449 +0000 UTC m=+10.040019185 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d-metrics-certs") pod "network-metrics-daemon-cmf6c" (UID: "a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 18:15:24.643450 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:24.643414 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/89de43e0-f41b-4570-af10-b3a3ef063480-original-pull-secret\") pod \"global-pull-secret-syncer-sm2rw\" (UID: \"89de43e0-f41b-4570-af10-b3a3ef063480\") " pod="kube-system/global-pull-secret-syncer-sm2rw" Sep 11 18:15:24.643683 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:24.643603 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 18:15:24.643683 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:24.643676 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89de43e0-f41b-4570-af10-b3a3ef063480-original-pull-secret podName:89de43e0-f41b-4570-af10-b3a3ef063480 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:28.64365641 +0000 UTC m=+10.140674159 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/89de43e0-f41b-4570-af10-b3a3ef063480-original-pull-secret") pod "global-pull-secret-syncer-sm2rw" (UID: "89de43e0-f41b-4570-af10-b3a3ef063480") : object "kube-system"/"original-pull-secret" not registered Sep 11 18:15:25.133542 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:25.133505 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cmf6c" Sep 11 18:15:25.133731 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:25.133653 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cmf6c" podUID="a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d" Sep 11 18:15:25.134101 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:25.134078 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm2rw" Sep 11 18:15:25.134201 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:25.134180 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm2rw" podUID="89de43e0-f41b-4570-af10-b3a3ef063480" Sep 11 18:15:25.134260 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:25.134242 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr7xc" Sep 11 18:15:25.134326 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:25.134307 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr7xc" podUID="e183b6fd-207a-44ed-829a-6872d65d0a57" Sep 11 18:15:27.134444 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:27.134067 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr7xc" Sep 11 18:15:27.134444 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:27.134207 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr7xc" podUID="e183b6fd-207a-44ed-829a-6872d65d0a57" Sep 11 18:15:27.134444 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:27.134067 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm2rw" Sep 11 18:15:27.134444 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:27.134255 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cmf6c" Sep 11 18:15:27.134444 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:27.134314 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm2rw" podUID="89de43e0-f41b-4570-af10-b3a3ef063480" Sep 11 18:15:27.134444 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:27.134397 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cmf6c" podUID="a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d" Sep 11 18:15:28.573128 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:28.573095 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d-metrics-certs\") pod \"network-metrics-daemon-cmf6c\" (UID: \"a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d\") " pod="openshift-multus/network-metrics-daemon-cmf6c" Sep 11 18:15:28.573654 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:28.573157 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g69tq\" (UniqueName: \"kubernetes.io/projected/e183b6fd-207a-44ed-829a-6872d65d0a57-kube-api-access-g69tq\") pod \"network-check-target-zr7xc\" (UID: \"e183b6fd-207a-44ed-829a-6872d65d0a57\") " pod="openshift-network-diagnostics/network-check-target-zr7xc" Sep 11 18:15:28.573654 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:28.573337 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 18:15:28.573654 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:28.573356 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 18:15:28.573654 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:28.573371 2561 projected.go:196] Error preparing data for projected volume kube-api-access-g69tq for pod openshift-network-diagnostics/network-check-target-zr7xc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 18:15:28.573654 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:28.573429 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e183b6fd-207a-44ed-829a-6872d65d0a57-kube-api-access-g69tq podName:e183b6fd-207a-44ed-829a-6872d65d0a57 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:36.573411866 +0000 UTC m=+18.070429619 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-g69tq" (UniqueName: "kubernetes.io/projected/e183b6fd-207a-44ed-829a-6872d65d0a57-kube-api-access-g69tq") pod "network-check-target-zr7xc" (UID: "e183b6fd-207a-44ed-829a-6872d65d0a57") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 18:15:28.573898 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:28.573820 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 18:15:28.573898 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:28.573874 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d-metrics-certs podName:a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d nodeName:}" failed. No retries permitted until 2026-09-11 18:15:36.573858642 +0000 UTC m=+18.070876397 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d-metrics-certs") pod "network-metrics-daemon-cmf6c" (UID: "a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 18:15:28.673769 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:28.673725 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/89de43e0-f41b-4570-af10-b3a3ef063480-original-pull-secret\") pod \"global-pull-secret-syncer-sm2rw\" (UID: \"89de43e0-f41b-4570-af10-b3a3ef063480\") " pod="kube-system/global-pull-secret-syncer-sm2rw" Sep 11 18:15:28.673968 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:28.673874 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 18:15:28.673968 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:28.673937 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89de43e0-f41b-4570-af10-b3a3ef063480-original-pull-secret podName:89de43e0-f41b-4570-af10-b3a3ef063480 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:36.673917715 +0000 UTC m=+18.170935461 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/89de43e0-f41b-4570-af10-b3a3ef063480-original-pull-secret") pod "global-pull-secret-syncer-sm2rw" (UID: "89de43e0-f41b-4570-af10-b3a3ef063480") : object "kube-system"/"original-pull-secret" not registered Sep 11 18:15:29.133894 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:29.133859 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cmf6c" Sep 11 18:15:29.134100 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:29.133963 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cmf6c" podUID="a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d" Sep 11 18:15:29.134100 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:29.133995 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr7xc" Sep 11 18:15:29.134100 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:29.133999 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm2rw" Sep 11 18:15:29.134273 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:29.134120 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr7xc" podUID="e183b6fd-207a-44ed-829a-6872d65d0a57" Sep 11 18:15:29.134273 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:29.134126 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm2rw" podUID="89de43e0-f41b-4570-af10-b3a3ef063480" Sep 11 18:15:31.133208 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:31.133172 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm2rw" Sep 11 18:15:31.133687 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:31.133213 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr7xc" Sep 11 18:15:31.133687 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:31.133172 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cmf6c" Sep 11 18:15:31.133687 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:31.133309 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm2rw" podUID="89de43e0-f41b-4570-af10-b3a3ef063480" Sep 11 18:15:31.133687 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:31.133410 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr7xc" podUID="e183b6fd-207a-44ed-829a-6872d65d0a57" Sep 11 18:15:31.133687 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:31.133521 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cmf6c" podUID="a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d" Sep 11 18:15:33.133218 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:33.133175 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm2rw" Sep 11 18:15:33.133630 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:33.133301 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm2rw" podUID="89de43e0-f41b-4570-af10-b3a3ef063480" Sep 11 18:15:33.133630 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:33.133175 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cmf6c" Sep 11 18:15:33.133630 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:33.133335 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr7xc" Sep 11 18:15:33.133630 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:33.133455 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cmf6c" podUID="a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d" Sep 11 18:15:33.133630 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:33.133538 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr7xc" podUID="e183b6fd-207a-44ed-829a-6872d65d0a57" Sep 11 18:15:35.132413 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:35.132375 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr7xc" Sep 11 18:15:35.132837 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:35.132378 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm2rw" Sep 11 18:15:35.132837 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:35.132511 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr7xc" podUID="e183b6fd-207a-44ed-829a-6872d65d0a57" Sep 11 18:15:35.132837 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:35.132557 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm2rw" podUID="89de43e0-f41b-4570-af10-b3a3ef063480" Sep 11 18:15:35.132837 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:35.132389 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cmf6c" Sep 11 18:15:35.132837 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:35.132651 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cmf6c" podUID="a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d" Sep 11 18:15:36.217978 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:36.217762 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fd24c" event={"ID":"3c0fffdb-93cd-4936-96d6-1b4ef7d2486c","Type":"ContainerStarted","Data":"cb3d026ef60143f3017cffcb5c2e77972542e499fedeede9c7e5e13d86a9d64d"} Sep 11 18:15:36.219477 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:36.219447 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gt5ln" event={"ID":"e9c4d02b-cc0d-4812-8fe2-80f5276923ba","Type":"ContainerStarted","Data":"848a14b05a8036f39bfc9461962794acc36a6cf44a4d8d59c13d2dcbefbc2134"} Sep 11 18:15:36.221523 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:36.221492 2561 generic.go:356] "Generic (PLEG): container finished" podID="4d1be3a0-f7d6-475b-b79f-c4fde3f4bcb9" containerID="8d60f29aed1a3b7318be851e97dd136b93d77b141889f5f8b04582f8616b4733" exitCode=0 Sep 11 18:15:36.221630 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:36.221588 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m9j9m" event={"ID":"4d1be3a0-f7d6-475b-b79f-c4fde3f4bcb9","Type":"ContainerDied","Data":"8d60f29aed1a3b7318be851e97dd136b93d77b141889f5f8b04582f8616b4733"} Sep 11 18:15:36.223985 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:36.223953 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-k7qjg" event={"ID":"6547cc6d-cb31-474a-8065-7fd47766fe02","Type":"ContainerStarted","Data":"171f12312c0a1f2bb6191bdc9bf8b098f87bb7ff282b32bdc588c606dd5510e9"} Sep 11 18:15:36.226085 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:36.226040 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-76xpz" event={"ID":"afbcd3d2-b5c0-474c-a177-e42b915d310d","Type":"ContainerStarted","Data":"3939844917ca79eb81579e4efe47265226bfbbc83328ae9fa8992ce6b989c69d"} Sep 11 18:15:36.227875 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:36.227848 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nwpdb" event={"ID":"72de70ec-e2ce-47b2-883d-75f4eb8eb39c","Type":"ContainerStarted","Data":"af6a33a12ca42eb917e664394d0019155822add11d35a45ca40b68a5f2e3360e"} Sep 11 18:15:36.229894 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:36.229868 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-128.ec2.internal" event={"ID":"ebf313d069610fccaa30accef842e07e","Type":"ContainerStarted","Data":"89147d87d79e33640497ead6e00b0f812bd69544b264462cd89234b00c4556b7"} Sep 11 18:15:36.231983 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:36.231963 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xh28n" event={"ID":"7f7d5659-22fb-43f1-8970-47837d126795","Type":"ContainerStarted","Data":"99d2aa56e67b1f26920c21a4b28e17daae29dfdf0bb664e9d80032e8aeffc488"} Sep 11 18:15:36.255459 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:36.255400 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-gt5ln" podStartSLOduration=3.020658854 podStartE2EDuration="17.255376364s" totalImagesPullingTime="14.23471751s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:15:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:36.234791166 +0000 UTC m=+17.731808922" watchObservedRunningTime="2026-09-11 18:15:36.255376364 +0000 UTC m=+17.752394119" Sep 11 18:15:36.269212 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:36.269164 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-128.ec2.internal" podStartSLOduration=17.269149427 podStartE2EDuration="17.269149427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:36.268740993 +0000 UTC m=+17.765758747" watchObservedRunningTime="2026-09-11 18:15:36.269149427 +0000 UTC m=+17.766167183" Sep 11 18:15:36.282890 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:36.282838 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-k7qjg" podStartSLOduration=3.075595024 podStartE2EDuration="17.28282463s" totalImagesPullingTime="14.207229606s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:36.282499511 +0000 UTC m=+17.779517269" watchObservedRunningTime="2026-09-11 18:15:36.28282463 +0000 UTC m=+17.779842384" Sep 11 18:15:36.299430 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:36.299383 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-nwpdb" podStartSLOduration=4.187941407 podStartE2EDuration="17.299368304s" totalImagesPullingTime="13.111426897s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:36.298349292 +0000 UTC m=+17.795367047" watchObservedRunningTime="2026-09-11 18:15:36.299368304 +0000 UTC m=+17.796386083" Sep 11 18:15:36.322267 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:36.322168 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-76xpz" podStartSLOduration=2.993510421 podStartE2EDuration="17.322147634s" totalImagesPullingTime="14.328637213s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:36.32197219 +0000 UTC m=+17.818989947" watchObservedRunningTime="2026-09-11 18:15:36.322147634 +0000 UTC m=+17.819165391" Sep 11 18:15:36.339569 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:36.339525 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-xh28n" podStartSLOduration=3.125932933 podStartE2EDuration="17.339508377s" totalImagesPullingTime="14.213575444s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:36.339176398 +0000 UTC m=+17.836194154" watchObservedRunningTime="2026-09-11 18:15:36.339508377 +0000 UTC m=+17.836526130" Sep 11 18:15:36.641135 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:36.641044 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g69tq\" (UniqueName: \"kubernetes.io/projected/e183b6fd-207a-44ed-829a-6872d65d0a57-kube-api-access-g69tq\") pod \"network-check-target-zr7xc\" (UID: \"e183b6fd-207a-44ed-829a-6872d65d0a57\") " pod="openshift-network-diagnostics/network-check-target-zr7xc" Sep 11 18:15:36.641135 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:36.641115 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d-metrics-certs\") pod \"network-metrics-daemon-cmf6c\" (UID: \"a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d\") " pod="openshift-multus/network-metrics-daemon-cmf6c" Sep 11 18:15:36.641422 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:36.641197 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 18:15:36.641422 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:36.641217 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 18:15:36.641422 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:36.641241 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 18:15:36.641422 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:36.641245 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d-metrics-certs podName:a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d nodeName:}" failed. No retries permitted until 2026-09-11 18:15:52.641233003 +0000 UTC m=+34.138250748 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d-metrics-certs") pod "network-metrics-daemon-cmf6c" (UID: "a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 18:15:36.641422 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:36.641252 2561 projected.go:196] Error preparing data for projected volume kube-api-access-g69tq for pod openshift-network-diagnostics/network-check-target-zr7xc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 18:15:36.641422 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:36.641316 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e183b6fd-207a-44ed-829a-6872d65d0a57-kube-api-access-g69tq podName:e183b6fd-207a-44ed-829a-6872d65d0a57 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:52.641287609 +0000 UTC m=+34.138305346 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-g69tq" (UniqueName: "kubernetes.io/projected/e183b6fd-207a-44ed-829a-6872d65d0a57-kube-api-access-g69tq") pod "network-check-target-zr7xc" (UID: "e183b6fd-207a-44ed-829a-6872d65d0a57") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 18:15:36.741668 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:36.741631 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/89de43e0-f41b-4570-af10-b3a3ef063480-original-pull-secret\") pod \"global-pull-secret-syncer-sm2rw\" (UID: \"89de43e0-f41b-4570-af10-b3a3ef063480\") " pod="kube-system/global-pull-secret-syncer-sm2rw" Sep 11 18:15:36.741830 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:36.741809 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 18:15:36.741888 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:36.741875 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89de43e0-f41b-4570-af10-b3a3ef063480-original-pull-secret podName:89de43e0-f41b-4570-af10-b3a3ef063480 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:52.741859984 +0000 UTC m=+34.238877716 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/89de43e0-f41b-4570-af10-b3a3ef063480-original-pull-secret") pod "global-pull-secret-syncer-sm2rw" (UID: "89de43e0-f41b-4570-af10-b3a3ef063480") : object "kube-system"/"original-pull-secret" not registered Sep 11 18:15:37.133234 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:37.133194 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm2rw" Sep 11 18:15:37.133423 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:37.133332 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm2rw" podUID="89de43e0-f41b-4570-af10-b3a3ef063480" Sep 11 18:15:37.133736 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:37.133708 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr7xc" Sep 11 18:15:37.133876 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:37.133857 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr7xc" podUID="e183b6fd-207a-44ed-829a-6872d65d0a57" Sep 11 18:15:37.134019 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:37.133996 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cmf6c" Sep 11 18:15:37.134170 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:37.134100 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cmf6c" podUID="a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d" Sep 11 18:15:37.236574 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:37.236466 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7b2vr" event={"ID":"b7bfcee0-0fd4-40c3-bf79-50ac2a7abb80","Type":"ContainerStarted","Data":"f28a3458ccc6b2e1cbf56e3e78dd7a757223c026d8a12d72c904ce6eb13d1d18"} Sep 11 18:15:37.263424 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:37.263364 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-7b2vr" podStartSLOduration=4.020775795 podStartE2EDuration="18.263349022s" totalImagesPullingTime="14.242573227s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:37.263195432 +0000 UTC m=+18.760213188" watchObservedRunningTime="2026-09-11 18:15:37.263349022 +0000 UTC m=+18.760366781" Sep 11 18:15:37.831203 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:37.831154 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-gt5ln" Sep 11 18:15:39.133377 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:39.133342 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cmf6c" Sep 11 18:15:39.133854 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:39.133472 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cmf6c" podUID="a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d" Sep 11 18:15:39.133854 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:39.133831 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm2rw" Sep 11 18:15:39.133973 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:39.133929 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm2rw" podUID="89de43e0-f41b-4570-af10-b3a3ef063480" Sep 11 18:15:39.134030 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:39.133976 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr7xc" Sep 11 18:15:39.134099 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:39.134037 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr7xc" podUID="e183b6fd-207a-44ed-829a-6872d65d0a57" Sep 11 18:15:41.061609 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:41.061252 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-gt5ln" Sep 11 18:15:41.062107 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:41.061961 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-gt5ln" Sep 11 18:15:41.132821 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:41.132789 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm2rw" Sep 11 18:15:41.132997 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:41.132789 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr7xc" Sep 11 18:15:41.132997 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:41.132925 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cmf6c" Sep 11 18:15:41.132997 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:41.132920 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm2rw" podUID="89de43e0-f41b-4570-af10-b3a3ef063480" Sep 11 18:15:41.133197 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:41.133005 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr7xc" podUID="e183b6fd-207a-44ed-829a-6872d65d0a57" Sep 11 18:15:41.133197 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:41.133128 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cmf6c" podUID="a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d" Sep 11 18:15:41.246201 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:41.246175 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-gt5ln" Sep 11 18:15:42.586148 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:42.586113 2561 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 11 18:15:43.103751 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:43.103515 2561 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T18:15:42.586131182Z","UUID":"e3ed7dfa-8af9-4846-af7a-f9a1244a0dd5","Handler":null,"Name":"","Endpoint":""} Sep 11 18:15:43.105297 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:43.105278 2561 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 11 18:15:43.105381 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:43.105309 2561 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 11 18:15:43.133097 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:43.133065 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cmf6c" Sep 11 18:15:43.133291 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:43.133210 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cmf6c" podUID="a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d" Sep 11 18:15:43.133573 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:43.133551 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm2rw" Sep 11 18:15:43.133698 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:43.133654 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm2rw" podUID="89de43e0-f41b-4570-af10-b3a3ef063480" Sep 11 18:15:43.133757 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:43.133715 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr7xc" Sep 11 18:15:43.133805 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:43.133780 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr7xc" podUID="e183b6fd-207a-44ed-829a-6872d65d0a57" Sep 11 18:15:43.252431 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:43.252340 2561 generic.go:356] "Generic (PLEG): container finished" podID="4d1be3a0-f7d6-475b-b79f-c4fde3f4bcb9" containerID="c4cbe2102bca294f86cb1c86442547e608a444826b543de7b066f6935bf700d2" exitCode=0 Sep 11 18:15:43.252573 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:43.252440 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m9j9m" event={"ID":"4d1be3a0-f7d6-475b-b79f-c4fde3f4bcb9","Type":"ContainerDied","Data":"c4cbe2102bca294f86cb1c86442547e608a444826b543de7b066f6935bf700d2"} Sep 11 18:15:43.255483 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:43.255300 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fd24c" event={"ID":"3c0fffdb-93cd-4936-96d6-1b4ef7d2486c","Type":"ContainerStarted","Data":"82a1ecbeff794e08cc1fd4b4eb8fd6ee670ffb21587907bfb612f714bcdc3470"} Sep 11 18:15:43.258972 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:43.258949 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4knd_42e5ddd8-a96c-45a1-adf9-caa47ee902db/ovn-acl-logging/0.log" Sep 11 18:15:43.259392 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:43.259374 2561 generic.go:356] "Generic (PLEG): container finished" podID="42e5ddd8-a96c-45a1-adf9-caa47ee902db" containerID="a0153028b21e90e679a96230407fc51dc7a1844d51a646385001a47d1a254994" exitCode=1 Sep 11 18:15:43.259466 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:43.259411 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" event={"ID":"42e5ddd8-a96c-45a1-adf9-caa47ee902db","Type":"ContainerStarted","Data":"e1d912f7b3a3a59a7b7295426c484fa5f012c56daa537efa5a74ef227df573a3"} Sep 11 18:15:43.259521 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:43.259484 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" event={"ID":"42e5ddd8-a96c-45a1-adf9-caa47ee902db","Type":"ContainerStarted","Data":"7220af25ef49a3564348d0e9e9db59f4e7f1bae4679329cde0f4f8622d20fdad"} Sep 11 18:15:43.259576 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:43.259532 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" event={"ID":"42e5ddd8-a96c-45a1-adf9-caa47ee902db","Type":"ContainerStarted","Data":"a683c5811951b968fab3d2becce15901e8d308935173b629a92d45f031d369c6"} Sep 11 18:15:43.259576 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:43.259565 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" event={"ID":"42e5ddd8-a96c-45a1-adf9-caa47ee902db","Type":"ContainerStarted","Data":"7a12c9d1c4f1fc74a6b29a609cd8b4658bfa98267edebee54d7302f0474d49c5"} Sep 11 18:15:43.259672 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:43.259600 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" event={"ID":"42e5ddd8-a96c-45a1-adf9-caa47ee902db","Type":"ContainerDied","Data":"a0153028b21e90e679a96230407fc51dc7a1844d51a646385001a47d1a254994"} Sep 11 18:15:43.259672 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:43.259640 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" event={"ID":"42e5ddd8-a96c-45a1-adf9-caa47ee902db","Type":"ContainerStarted","Data":"230aef4570bad23512f751c84b887bf0176cfbcb4baa7fd52819762a676aaed8"} Sep 11 18:15:44.263001 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:44.262964 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fd24c" event={"ID":"3c0fffdb-93cd-4936-96d6-1b4ef7d2486c","Type":"ContainerStarted","Data":"16e8e2102ad85cbb601968620ed6ae0339e2e20fb053db6d61dba1f08ca5eb72"} Sep 11 18:15:44.280344 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:44.280293 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fd24c" podStartSLOduration=3.28451268 podStartE2EDuration="25.280276031s" totalImagesPullingTime="21.995763351s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:44.280252079 +0000 UTC m=+25.777269833" watchObservedRunningTime="2026-09-11 18:15:44.280276031 +0000 UTC m=+25.777293776" Sep 11 18:15:45.135677 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:45.135642 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cmf6c" Sep 11 18:15:45.135860 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:45.135646 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm2rw" Sep 11 18:15:45.135860 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:45.135765 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cmf6c" podUID="a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d" Sep 11 18:15:45.135860 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:45.135646 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr7xc" Sep 11 18:15:45.135860 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:45.135821 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm2rw" podUID="89de43e0-f41b-4570-af10-b3a3ef063480" Sep 11 18:15:45.136079 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:45.135876 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr7xc" podUID="e183b6fd-207a-44ed-829a-6872d65d0a57" Sep 11 18:15:45.270988 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:45.270963 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4knd_42e5ddd8-a96c-45a1-adf9-caa47ee902db/ovn-acl-logging/0.log" Sep 11 18:15:45.271430 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:45.271397 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" event={"ID":"42e5ddd8-a96c-45a1-adf9-caa47ee902db","Type":"ContainerStarted","Data":"471f5f1d37aaab31b0f9f47c7a5cbe971c4205368679338c9bbb974f46925867"} Sep 11 18:15:45.273606 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:45.273575 2561 generic.go:356] "Generic (PLEG): container finished" podID="4d1be3a0-f7d6-475b-b79f-c4fde3f4bcb9" containerID="865fbb1439418eb9dafe8a15c20a2717e177ccb5fdc711d81866bf9c2bba395c" exitCode=0 Sep 11 18:15:45.273703 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:45.273646 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m9j9m" event={"ID":"4d1be3a0-f7d6-475b-b79f-c4fde3f4bcb9","Type":"ContainerDied","Data":"865fbb1439418eb9dafe8a15c20a2717e177ccb5fdc711d81866bf9c2bba395c"} Sep 11 18:15:46.278860 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:46.278771 2561 generic.go:356] "Generic (PLEG): container finished" podID="4d1be3a0-f7d6-475b-b79f-c4fde3f4bcb9" containerID="ac3ed786cb1463baec2ecb5366da0c72dedd15f706fde9dd7c792f5669162c19" exitCode=0 Sep 11 18:15:46.278860 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:46.278821 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m9j9m" event={"ID":"4d1be3a0-f7d6-475b-b79f-c4fde3f4bcb9","Type":"ContainerDied","Data":"ac3ed786cb1463baec2ecb5366da0c72dedd15f706fde9dd7c792f5669162c19"} Sep 11 18:15:47.132638 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:47.132603 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm2rw" Sep 11 18:15:47.132817 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:47.132609 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr7xc" Sep 11 18:15:47.132817 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:47.132713 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cmf6c" Sep 11 18:15:47.132817 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:47.132763 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm2rw" podUID="89de43e0-f41b-4570-af10-b3a3ef063480" Sep 11 18:15:47.132817 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:47.132771 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr7xc" podUID="e183b6fd-207a-44ed-829a-6872d65d0a57" Sep 11 18:15:47.132817 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:47.132812 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cmf6c" podUID="a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d" Sep 11 18:15:48.287373 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:48.287338 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4knd_42e5ddd8-a96c-45a1-adf9-caa47ee902db/ovn-acl-logging/0.log" Sep 11 18:15:48.288144 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:48.287723 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" event={"ID":"42e5ddd8-a96c-45a1-adf9-caa47ee902db","Type":"ContainerStarted","Data":"373e45b2bcb6698b2ffd8336fb44bab201cb0353e9cfdf1d6ef76014fc0b910d"} Sep 11 18:15:48.288144 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:48.288048 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:48.288308 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:48.288291 2561 scope.go:117] "RemoveContainer" containerID="a0153028b21e90e679a96230407fc51dc7a1844d51a646385001a47d1a254994" Sep 11 18:15:48.306226 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:48.306193 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:49.133561 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:49.133355 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cmf6c" Sep 11 18:15:49.133726 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:49.133401 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm2rw" Sep 11 18:15:49.133726 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:49.133650 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cmf6c" podUID="a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d" Sep 11 18:15:49.133835 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:49.133734 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm2rw" podUID="89de43e0-f41b-4570-af10-b3a3ef063480" Sep 11 18:15:49.133835 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:49.133432 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr7xc" Sep 11 18:15:49.133954 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:49.133859 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr7xc" podUID="e183b6fd-207a-44ed-829a-6872d65d0a57" Sep 11 18:15:49.242078 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:49.241974 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sm2rw"] Sep 11 18:15:49.249788 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:49.249751 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cmf6c"] Sep 11 18:15:49.250412 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:49.250370 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zr7xc"] Sep 11 18:15:49.295917 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:49.295806 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4knd_42e5ddd8-a96c-45a1-adf9-caa47ee902db/ovn-acl-logging/0.log" Sep 11 18:15:49.296496 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:49.296272 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" event={"ID":"42e5ddd8-a96c-45a1-adf9-caa47ee902db","Type":"ContainerStarted","Data":"bf7d2e80cf06457d4bf0d50175d14d3e02f599ff6a06b170d1ff532359aff45d"} Sep 11 18:15:49.296496 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:49.296352 2561 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 11 18:15:49.296496 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:49.296398 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr7xc" Sep 11 18:15:49.296652 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:49.296589 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr7xc" podUID="e183b6fd-207a-44ed-829a-6872d65d0a57" Sep 11 18:15:49.296652 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:49.296610 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm2rw" Sep 11 18:15:49.296755 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:49.296696 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm2rw" podUID="89de43e0-f41b-4570-af10-b3a3ef063480" Sep 11 18:15:49.296755 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:49.296748 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:49.296854 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:49.296805 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cmf6c" Sep 11 18:15:49.296912 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:49.296897 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cmf6c" podUID="a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d" Sep 11 18:15:49.314182 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:49.314153 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:49.325223 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:49.325178 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" podStartSLOduration=9.094416458 podStartE2EDuration="30.325163191s" totalImagesPullingTime="21.230746733s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:49.324196038 +0000 UTC m=+30.821213792" watchObservedRunningTime="2026-09-11 18:15:49.325163191 +0000 UTC m=+30.822180944" Sep 11 18:15:50.178271 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:50.178221 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:15:51.132674 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:51.132640 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cmf6c" Sep 11 18:15:51.133138 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:51.132640 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm2rw" Sep 11 18:15:51.133138 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:51.132645 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr7xc" Sep 11 18:15:51.133138 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:51.132824 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm2rw" podUID="89de43e0-f41b-4570-af10-b3a3ef063480" Sep 11 18:15:51.133138 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:51.132899 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr7xc" podUID="e183b6fd-207a-44ed-829a-6872d65d0a57" Sep 11 18:15:51.133138 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:51.132812 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cmf6c" podUID="a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d" Sep 11 18:15:51.304345 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:51.304310 2561 generic.go:356] "Generic (PLEG): container finished" podID="4d1be3a0-f7d6-475b-b79f-c4fde3f4bcb9" containerID="bdfbc468d8954a0369d11d752e19cbe3e30197ca577c2977f406bdabca2d0684" exitCode=0 Sep 11 18:15:51.304489 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:51.304440 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m9j9m" event={"ID":"4d1be3a0-f7d6-475b-b79f-c4fde3f4bcb9","Type":"ContainerDied","Data":"bdfbc468d8954a0369d11d752e19cbe3e30197ca577c2977f406bdabca2d0684"} Sep 11 18:15:52.310092 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:52.309989 2561 generic.go:356] "Generic (PLEG): container finished" podID="4d1be3a0-f7d6-475b-b79f-c4fde3f4bcb9" containerID="051ad104f5cb24bd92a0df6ac6640271bad84aba0619af604f1a63820b2d9a20" exitCode=0 Sep 11 18:15:52.310092 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:52.310037 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m9j9m" event={"ID":"4d1be3a0-f7d6-475b-b79f-c4fde3f4bcb9","Type":"ContainerDied","Data":"051ad104f5cb24bd92a0df6ac6640271bad84aba0619af604f1a63820b2d9a20"} Sep 11 18:15:52.657488 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:52.657458 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d-metrics-certs\") pod \"network-metrics-daemon-cmf6c\" (UID: \"a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d\") " pod="openshift-multus/network-metrics-daemon-cmf6c" Sep 11 18:15:52.657648 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:52.657505 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g69tq\" (UniqueName: \"kubernetes.io/projected/e183b6fd-207a-44ed-829a-6872d65d0a57-kube-api-access-g69tq\") pod \"network-check-target-zr7xc\" (UID: \"e183b6fd-207a-44ed-829a-6872d65d0a57\") " pod="openshift-network-diagnostics/network-check-target-zr7xc" Sep 11 18:15:52.657648 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:52.657610 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 18:15:52.657715 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:52.657683 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d-metrics-certs podName:a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d nodeName:}" failed. No retries permitted until 2026-09-11 18:16:24.657666585 +0000 UTC m=+66.154684317 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d-metrics-certs") pod "network-metrics-daemon-cmf6c" (UID: "a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 18:15:52.657715 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:52.657616 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 18:15:52.657786 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:52.657719 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 18:15:52.657786 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:52.657732 2561 projected.go:196] Error preparing data for projected volume kube-api-access-g69tq for pod openshift-network-diagnostics/network-check-target-zr7xc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 18:15:52.657786 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:52.657781 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e183b6fd-207a-44ed-829a-6872d65d0a57-kube-api-access-g69tq podName:e183b6fd-207a-44ed-829a-6872d65d0a57 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:24.657766751 +0000 UTC m=+66.154784482 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-g69tq" (UniqueName: "kubernetes.io/projected/e183b6fd-207a-44ed-829a-6872d65d0a57-kube-api-access-g69tq") pod "network-check-target-zr7xc" (UID: "e183b6fd-207a-44ed-829a-6872d65d0a57") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 18:15:52.758524 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:52.758490 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/89de43e0-f41b-4570-af10-b3a3ef063480-original-pull-secret\") pod \"global-pull-secret-syncer-sm2rw\" (UID: \"89de43e0-f41b-4570-af10-b3a3ef063480\") " pod="kube-system/global-pull-secret-syncer-sm2rw" Sep 11 18:15:52.758667 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:52.758637 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 18:15:52.758724 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:52.758704 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89de43e0-f41b-4570-af10-b3a3ef063480-original-pull-secret podName:89de43e0-f41b-4570-af10-b3a3ef063480 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:24.75868769 +0000 UTC m=+66.255705429 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/89de43e0-f41b-4570-af10-b3a3ef063480-original-pull-secret") pod "global-pull-secret-syncer-sm2rw" (UID: "89de43e0-f41b-4570-af10-b3a3ef063480") : object "kube-system"/"original-pull-secret" not registered Sep 11 18:15:53.133571 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:53.133487 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr7xc" Sep 11 18:15:53.133752 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:53.133487 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm2rw" Sep 11 18:15:53.133752 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:53.133605 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cmf6c" Sep 11 18:15:53.133752 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:53.133603 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr7xc" podUID="e183b6fd-207a-44ed-829a-6872d65d0a57" Sep 11 18:15:53.133752 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:53.133674 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm2rw" podUID="89de43e0-f41b-4570-af10-b3a3ef063480" Sep 11 18:15:53.133959 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:53.133767 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cmf6c" podUID="a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d" Sep 11 18:15:53.315624 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:53.315389 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m9j9m" event={"ID":"4d1be3a0-f7d6-475b-b79f-c4fde3f4bcb9","Type":"ContainerStarted","Data":"f84d9021eb64097ee83cf994f98f4489d304b386b146682ff5e8ad7f11e959ef"} Sep 11 18:15:53.336920 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:53.336871 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-m9j9m" podStartSLOduration=9.240435386 podStartE2EDuration="34.336854936s" totalImagesPullingTime="25.09641955s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:19 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:53.336834293 +0000 UTC m=+34.833852049" watchObservedRunningTime="2026-09-11 18:15:53.336854936 +0000 UTC m=+34.833872682" Sep 11 18:15:55.135557 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:55.135531 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cmf6c" Sep 11 18:15:55.135970 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:55.135531 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm2rw" Sep 11 18:15:55.135970 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:55.135637 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cmf6c" podUID="a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d" Sep 11 18:15:55.135970 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:55.135697 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm2rw" podUID="89de43e0-f41b-4570-af10-b3a3ef063480" Sep 11 18:15:55.135970 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:55.135531 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr7xc" Sep 11 18:15:55.135970 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:55.135769 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr7xc" podUID="e183b6fd-207a-44ed-829a-6872d65d0a57" Sep 11 18:15:55.230117 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:55.230089 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-128.ec2.internal" event="NodeReady" Sep 11 18:15:55.230248 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:55.230235 2561 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 11 18:15:55.285585 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:55.285553 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-c6nk8"] Sep 11 18:15:55.298459 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:55.298427 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-x6rnn"] Sep 11 18:15:55.298613 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:55.298554 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c6nk8" Sep 11 18:15:55.301591 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:55.301562 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-b9ltz\"" Sep 11 18:15:55.301591 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:55.301585 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 11 18:15:55.301806 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:55.301586 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 11 18:15:55.308498 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:55.308476 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x6rnn"] Sep 11 18:15:55.308597 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:55.308507 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-c6nk8"] Sep 11 18:15:55.308633 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:55.308608 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x6rnn" Sep 11 18:15:55.310819 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:55.310803 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 11 18:15:55.310991 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:55.310969 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-kjt2v\"" Sep 11 18:15:55.311214 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:55.311197 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 11 18:15:55.311380 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:55.311365 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 11 18:15:55.373716 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:55.373693 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km6wn\" (UniqueName: \"kubernetes.io/projected/2c3feaeb-94a3-40e2-b158-976547caf818-kube-api-access-km6wn\") pod \"ingress-canary-x6rnn\" (UID: \"2c3feaeb-94a3-40e2-b158-976547caf818\") " pod="openshift-ingress-canary/ingress-canary-x6rnn" Sep 11 18:15:55.373836 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:55.373726 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/71d0b928-d82f-4a8d-844c-36613596b07b-config-volume\") pod \"dns-default-c6nk8\" (UID: \"71d0b928-d82f-4a8d-844c-36613596b07b\") " pod="openshift-dns/dns-default-c6nk8" Sep 11 18:15:55.373836 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:55.373761 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2c3feaeb-94a3-40e2-b158-976547caf818-cert\") pod \"ingress-canary-x6rnn\" (UID: \"2c3feaeb-94a3-40e2-b158-976547caf818\") " pod="openshift-ingress-canary/ingress-canary-x6rnn" Sep 11 18:15:55.373836 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:55.373778 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71d0b928-d82f-4a8d-844c-36613596b07b-tmp-dir\") pod \"dns-default-c6nk8\" (UID: \"71d0b928-d82f-4a8d-844c-36613596b07b\") " pod="openshift-dns/dns-default-c6nk8" Sep 11 18:15:55.373836 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:55.373792 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgbtv\" (UniqueName: \"kubernetes.io/projected/71d0b928-d82f-4a8d-844c-36613596b07b-kube-api-access-zgbtv\") pod \"dns-default-c6nk8\" (UID: \"71d0b928-d82f-4a8d-844c-36613596b07b\") " pod="openshift-dns/dns-default-c6nk8" Sep 11 18:15:55.373836 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:55.373810 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71d0b928-d82f-4a8d-844c-36613596b07b-metrics-tls\") pod \"dns-default-c6nk8\" (UID: \"71d0b928-d82f-4a8d-844c-36613596b07b\") " pod="openshift-dns/dns-default-c6nk8" Sep 11 18:15:55.475071 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:55.475013 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2c3feaeb-94a3-40e2-b158-976547caf818-cert\") pod \"ingress-canary-x6rnn\" (UID: \"2c3feaeb-94a3-40e2-b158-976547caf818\") " pod="openshift-ingress-canary/ingress-canary-x6rnn" Sep 11 18:15:55.475248 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:55.475078 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71d0b928-d82f-4a8d-844c-36613596b07b-tmp-dir\") pod \"dns-default-c6nk8\" (UID: \"71d0b928-d82f-4a8d-844c-36613596b07b\") " pod="openshift-dns/dns-default-c6nk8" Sep 11 18:15:55.475248 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:55.475104 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgbtv\" (UniqueName: \"kubernetes.io/projected/71d0b928-d82f-4a8d-844c-36613596b07b-kube-api-access-zgbtv\") pod \"dns-default-c6nk8\" (UID: \"71d0b928-d82f-4a8d-844c-36613596b07b\") " pod="openshift-dns/dns-default-c6nk8" Sep 11 18:15:55.475248 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:55.475127 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71d0b928-d82f-4a8d-844c-36613596b07b-metrics-tls\") pod \"dns-default-c6nk8\" (UID: \"71d0b928-d82f-4a8d-844c-36613596b07b\") " pod="openshift-dns/dns-default-c6nk8" Sep 11 18:15:55.475248 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:55.475230 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km6wn\" (UniqueName: \"kubernetes.io/projected/2c3feaeb-94a3-40e2-b158-976547caf818-kube-api-access-km6wn\") pod \"ingress-canary-x6rnn\" (UID: \"2c3feaeb-94a3-40e2-b158-976547caf818\") " pod="openshift-ingress-canary/ingress-canary-x6rnn" Sep 11 18:15:55.475458 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:55.475257 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/71d0b928-d82f-4a8d-844c-36613596b07b-config-volume\") pod \"dns-default-c6nk8\" (UID: \"71d0b928-d82f-4a8d-844c-36613596b07b\") " pod="openshift-dns/dns-default-c6nk8" Sep 11 18:15:55.475458 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:55.475396 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 18:15:55.475609 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:55.475464 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71d0b928-d82f-4a8d-844c-36613596b07b-metrics-tls podName:71d0b928-d82f-4a8d-844c-36613596b07b nodeName:}" failed. No retries permitted until 2026-09-11 18:15:55.975445454 +0000 UTC m=+37.472463201 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/71d0b928-d82f-4a8d-844c-36613596b07b-metrics-tls") pod "dns-default-c6nk8" (UID: "71d0b928-d82f-4a8d-844c-36613596b07b") : secret "dns-default-metrics-tls" not found Sep 11 18:15:55.475609 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:55.475527 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71d0b928-d82f-4a8d-844c-36613596b07b-tmp-dir\") pod \"dns-default-c6nk8\" (UID: \"71d0b928-d82f-4a8d-844c-36613596b07b\") " pod="openshift-dns/dns-default-c6nk8" Sep 11 18:15:55.475778 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:55.475761 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/71d0b928-d82f-4a8d-844c-36613596b07b-config-volume\") pod \"dns-default-c6nk8\" (UID: \"71d0b928-d82f-4a8d-844c-36613596b07b\") " pod="openshift-dns/dns-default-c6nk8" Sep 11 18:15:55.479309 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:55.479293 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2c3feaeb-94a3-40e2-b158-976547caf818-cert\") pod \"ingress-canary-x6rnn\" (UID: \"2c3feaeb-94a3-40e2-b158-976547caf818\") " pod="openshift-ingress-canary/ingress-canary-x6rnn" Sep 11 18:15:55.485929 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:55.485901 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgbtv\" (UniqueName: \"kubernetes.io/projected/71d0b928-d82f-4a8d-844c-36613596b07b-kube-api-access-zgbtv\") pod \"dns-default-c6nk8\" (UID: \"71d0b928-d82f-4a8d-844c-36613596b07b\") " pod="openshift-dns/dns-default-c6nk8" Sep 11 18:15:55.486037 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:55.485995 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-km6wn\" (UniqueName: \"kubernetes.io/projected/2c3feaeb-94a3-40e2-b158-976547caf818-kube-api-access-km6wn\") pod \"ingress-canary-x6rnn\" (UID: \"2c3feaeb-94a3-40e2-b158-976547caf818\") " pod="openshift-ingress-canary/ingress-canary-x6rnn" Sep 11 18:15:55.616260 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:55.616220 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x6rnn" Sep 11 18:15:55.923833 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:55.923804 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x6rnn"] Sep 11 18:15:55.927087 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:15:55.927046 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c3feaeb_94a3_40e2_b158_976547caf818.slice/crio-873a804e8ca1c7dced76f1a12b81524030def6321cda047a527070cbbec57fce WatchSource:0}: Error finding container 873a804e8ca1c7dced76f1a12b81524030def6321cda047a527070cbbec57fce: Status 404 returned error can't find the container with id 873a804e8ca1c7dced76f1a12b81524030def6321cda047a527070cbbec57fce Sep 11 18:15:55.978790 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:55.978762 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71d0b928-d82f-4a8d-844c-36613596b07b-metrics-tls\") pod \"dns-default-c6nk8\" (UID: \"71d0b928-d82f-4a8d-844c-36613596b07b\") " pod="openshift-dns/dns-default-c6nk8" Sep 11 18:15:55.978942 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:55.978892 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 18:15:55.979017 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:55.978951 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71d0b928-d82f-4a8d-844c-36613596b07b-metrics-tls podName:71d0b928-d82f-4a8d-844c-36613596b07b nodeName:}" failed. No retries permitted until 2026-09-11 18:15:56.97893876 +0000 UTC m=+38.475956492 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/71d0b928-d82f-4a8d-844c-36613596b07b-metrics-tls") pod "dns-default-c6nk8" (UID: "71d0b928-d82f-4a8d-844c-36613596b07b") : secret "dns-default-metrics-tls" not found Sep 11 18:15:56.322661 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:56.322579 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x6rnn" event={"ID":"2c3feaeb-94a3-40e2-b158-976547caf818","Type":"ContainerStarted","Data":"873a804e8ca1c7dced76f1a12b81524030def6321cda047a527070cbbec57fce"} Sep 11 18:15:56.986399 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:56.986367 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71d0b928-d82f-4a8d-844c-36613596b07b-metrics-tls\") pod \"dns-default-c6nk8\" (UID: \"71d0b928-d82f-4a8d-844c-36613596b07b\") " pod="openshift-dns/dns-default-c6nk8" Sep 11 18:15:56.986594 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:56.986514 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 18:15:56.986594 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:56.986577 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71d0b928-d82f-4a8d-844c-36613596b07b-metrics-tls podName:71d0b928-d82f-4a8d-844c-36613596b07b nodeName:}" failed. No retries permitted until 2026-09-11 18:15:58.986559071 +0000 UTC m=+40.483576818 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/71d0b928-d82f-4a8d-844c-36613596b07b-metrics-tls") pod "dns-default-c6nk8" (UID: "71d0b928-d82f-4a8d-844c-36613596b07b") : secret "dns-default-metrics-tls" not found Sep 11 18:15:57.133185 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:57.133146 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cmf6c" Sep 11 18:15:57.133373 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:57.133146 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr7xc" Sep 11 18:15:57.133438 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:57.133146 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm2rw" Sep 11 18:15:57.135789 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:57.135764 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 11 18:15:57.136662 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:57.136638 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 11 18:15:57.136779 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:57.136667 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-78r7v\"" Sep 11 18:15:57.136779 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:57.136669 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 11 18:15:57.136779 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:57.136669 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 11 18:15:57.136779 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:57.136671 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-28rql\"" Sep 11 18:15:59.003001 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:59.002815 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71d0b928-d82f-4a8d-844c-36613596b07b-metrics-tls\") pod \"dns-default-c6nk8\" (UID: \"71d0b928-d82f-4a8d-844c-36613596b07b\") " pod="openshift-dns/dns-default-c6nk8" Sep 11 18:15:59.003001 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:59.002959 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 18:15:59.003398 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:15:59.003080 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71d0b928-d82f-4a8d-844c-36613596b07b-metrics-tls podName:71d0b928-d82f-4a8d-844c-36613596b07b nodeName:}" failed. No retries permitted until 2026-09-11 18:16:03.003044991 +0000 UTC m=+44.500062723 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/71d0b928-d82f-4a8d-844c-36613596b07b-metrics-tls") pod "dns-default-c6nk8" (UID: "71d0b928-d82f-4a8d-844c-36613596b07b") : secret "dns-default-metrics-tls" not found Sep 11 18:15:59.330565 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:59.330531 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x6rnn" event={"ID":"2c3feaeb-94a3-40e2-b158-976547caf818","Type":"ContainerStarted","Data":"0910e7291299dc2284867fdcdb3b32caabf22c50ab9cc256a597d90a8af1c2cb"} Sep 11 18:15:59.347189 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:15:59.347128 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-x6rnn" podStartSLOduration=1.7967314399999998 podStartE2EDuration="4.347111007s" totalImagesPullingTime="2.550379567s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:15:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:59.345525387 +0000 UTC m=+40.842543153" watchObservedRunningTime="2026-09-11 18:15:59.347111007 +0000 UTC m=+40.844128760" Sep 11 18:16:03.031501 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:03.031461 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71d0b928-d82f-4a8d-844c-36613596b07b-metrics-tls\") pod \"dns-default-c6nk8\" (UID: \"71d0b928-d82f-4a8d-844c-36613596b07b\") " pod="openshift-dns/dns-default-c6nk8" Sep 11 18:16:03.031876 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:16:03.031623 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 18:16:03.031876 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:16:03.031706 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71d0b928-d82f-4a8d-844c-36613596b07b-metrics-tls podName:71d0b928-d82f-4a8d-844c-36613596b07b nodeName:}" failed. No retries permitted until 2026-09-11 18:16:11.031687734 +0000 UTC m=+52.528705470 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/71d0b928-d82f-4a8d-844c-36613596b07b-metrics-tls") pod "dns-default-c6nk8" (UID: "71d0b928-d82f-4a8d-844c-36613596b07b") : secret "dns-default-metrics-tls" not found Sep 11 18:16:11.094852 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:11.094811 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71d0b928-d82f-4a8d-844c-36613596b07b-metrics-tls\") pod \"dns-default-c6nk8\" (UID: \"71d0b928-d82f-4a8d-844c-36613596b07b\") " pod="openshift-dns/dns-default-c6nk8" Sep 11 18:16:11.095411 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:16:11.094962 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 18:16:11.095411 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:16:11.095028 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71d0b928-d82f-4a8d-844c-36613596b07b-metrics-tls podName:71d0b928-d82f-4a8d-844c-36613596b07b nodeName:}" failed. No retries permitted until 2026-09-11 18:16:27.095012895 +0000 UTC m=+68.592030643 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/71d0b928-d82f-4a8d-844c-36613596b07b-metrics-tls") pod "dns-default-c6nk8" (UID: "71d0b928-d82f-4a8d-844c-36613596b07b") : secret "dns-default-metrics-tls" not found Sep 11 18:16:21.321702 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:21.321674 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v4knd" Sep 11 18:16:24.694528 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:24.694483 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d-metrics-certs\") pod \"network-metrics-daemon-cmf6c\" (UID: \"a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d\") " pod="openshift-multus/network-metrics-daemon-cmf6c" Sep 11 18:16:24.694979 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:24.694543 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g69tq\" (UniqueName: \"kubernetes.io/projected/e183b6fd-207a-44ed-829a-6872d65d0a57-kube-api-access-g69tq\") pod \"network-check-target-zr7xc\" (UID: \"e183b6fd-207a-44ed-829a-6872d65d0a57\") " pod="openshift-network-diagnostics/network-check-target-zr7xc" Sep 11 18:16:24.697117 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:24.697090 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 11 18:16:24.697222 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:24.697154 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 11 18:16:24.704950 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:16:24.704930 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 11 18:16:24.705042 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:16:24.704989 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d-metrics-certs podName:a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d nodeName:}" failed. No retries permitted until 2026-09-11 18:17:28.704973585 +0000 UTC m=+130.201991317 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d-metrics-certs") pod "network-metrics-daemon-cmf6c" (UID: "a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d") : secret "metrics-daemon-secret" not found Sep 11 18:16:24.706758 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:24.706741 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 11 18:16:24.718835 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:24.718803 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g69tq\" (UniqueName: \"kubernetes.io/projected/e183b6fd-207a-44ed-829a-6872d65d0a57-kube-api-access-g69tq\") pod \"network-check-target-zr7xc\" (UID: \"e183b6fd-207a-44ed-829a-6872d65d0a57\") " pod="openshift-network-diagnostics/network-check-target-zr7xc" Sep 11 18:16:24.755338 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:24.755310 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-78r7v\"" Sep 11 18:16:24.763225 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:24.763198 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr7xc" Sep 11 18:16:24.794990 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:24.794955 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/89de43e0-f41b-4570-af10-b3a3ef063480-original-pull-secret\") pod \"global-pull-secret-syncer-sm2rw\" (UID: \"89de43e0-f41b-4570-af10-b3a3ef063480\") " pod="kube-system/global-pull-secret-syncer-sm2rw" Sep 11 18:16:24.797616 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:24.797592 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 11 18:16:24.807600 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:24.807569 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/89de43e0-f41b-4570-af10-b3a3ef063480-original-pull-secret\") pod \"global-pull-secret-syncer-sm2rw\" (UID: \"89de43e0-f41b-4570-af10-b3a3ef063480\") " pod="kube-system/global-pull-secret-syncer-sm2rw" Sep 11 18:16:25.032848 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:25.032762 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zr7xc"] Sep 11 18:16:25.036937 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:16:25.036894 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode183b6fd_207a_44ed_829a_6872d65d0a57.slice/crio-c5718cdb2688abee56242b317f4f3fb4d00d5058950b548e0b9f990c3172683b WatchSource:0}: Error finding container c5718cdb2688abee56242b317f4f3fb4d00d5058950b548e0b9f990c3172683b: Status 404 returned error can't find the container with id c5718cdb2688abee56242b317f4f3fb4d00d5058950b548e0b9f990c3172683b Sep 11 18:16:25.059592 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:25.059569 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm2rw" Sep 11 18:16:25.326423 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:25.326354 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sm2rw"] Sep 11 18:16:25.329811 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:16:25.329787 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89de43e0_f41b_4570_af10_b3a3ef063480.slice/crio-e307a0599c4750c877b9bc1a2d0102f1a3a2447500fac366d6da6b7746932041 WatchSource:0}: Error finding container e307a0599c4750c877b9bc1a2d0102f1a3a2447500fac366d6da6b7746932041: Status 404 returned error can't find the container with id e307a0599c4750c877b9bc1a2d0102f1a3a2447500fac366d6da6b7746932041 Sep 11 18:16:25.397557 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:25.397523 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sm2rw" event={"ID":"89de43e0-f41b-4570-af10-b3a3ef063480","Type":"ContainerStarted","Data":"e307a0599c4750c877b9bc1a2d0102f1a3a2447500fac366d6da6b7746932041"} Sep 11 18:16:25.398944 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:25.398914 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zr7xc" event={"ID":"e183b6fd-207a-44ed-829a-6872d65d0a57","Type":"ContainerStarted","Data":"c5718cdb2688abee56242b317f4f3fb4d00d5058950b548e0b9f990c3172683b"} Sep 11 18:16:27.108754 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:27.108715 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71d0b928-d82f-4a8d-844c-36613596b07b-metrics-tls\") pod \"dns-default-c6nk8\" (UID: \"71d0b928-d82f-4a8d-844c-36613596b07b\") " pod="openshift-dns/dns-default-c6nk8" Sep 11 18:16:27.109238 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:16:27.108896 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 18:16:27.109238 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:16:27.108981 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71d0b928-d82f-4a8d-844c-36613596b07b-metrics-tls podName:71d0b928-d82f-4a8d-844c-36613596b07b nodeName:}" failed. No retries permitted until 2026-09-11 18:16:59.108958942 +0000 UTC m=+100.605976678 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/71d0b928-d82f-4a8d-844c-36613596b07b-metrics-tls") pod "dns-default-c6nk8" (UID: "71d0b928-d82f-4a8d-844c-36613596b07b") : secret "dns-default-metrics-tls" not found Sep 11 18:16:28.409838 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:28.409803 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zr7xc" event={"ID":"e183b6fd-207a-44ed-829a-6872d65d0a57","Type":"ContainerStarted","Data":"c95d89b53ccab8879456d31446b4f4c8afe4ae33869f8eaa2138ae30b37b6d7d"} Sep 11 18:16:28.410282 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:28.409881 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zr7xc" Sep 11 18:16:28.429529 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:28.429477 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zr7xc" podStartSLOduration=66.363683149 podStartE2EDuration="1m9.429459261s" totalImagesPullingTime="3.065776112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:15:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:28.428683943 +0000 UTC m=+69.925701700" watchObservedRunningTime="2026-09-11 18:16:28.429459261 +0000 UTC m=+69.926477017" Sep 11 18:16:30.415525 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:30.415482 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sm2rw" event={"ID":"89de43e0-f41b-4570-af10-b3a3ef063480","Type":"ContainerStarted","Data":"ad7337b5ffff7a91c231600bcca0ec8f9cd3257788ce60addfc1a6fbf1974e9c"} Sep 11 18:16:30.432078 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:30.431901 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-sm2rw" podStartSLOduration=67.108475306 podStartE2EDuration="1m11.43188191s" totalImagesPullingTime="4.323406604s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:30.431691514 +0000 UTC m=+71.928709269" watchObservedRunningTime="2026-09-11 18:16:30.43188191 +0000 UTC m=+71.928899645" Sep 11 18:16:35.224666 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.224630 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-s6nf9"] Sep 11 18:16:35.226468 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.226450 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-s6nf9" Sep 11 18:16:35.231589 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.231559 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 11 18:16:35.231589 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.231574 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 11 18:16:35.231755 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.231574 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 11 18:16:35.231755 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.231567 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 11 18:16:35.240921 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.240903 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-mmd2n\"" Sep 11 18:16:35.265934 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.265908 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3a406a87-d961-4352-b049-eb5476b6f7a5-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-s6nf9\" (UID: \"3a406a87-d961-4352-b049-eb5476b6f7a5\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-s6nf9" Sep 11 18:16:35.266023 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.265949 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3a406a87-d961-4352-b049-eb5476b6f7a5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-s6nf9\" (UID: \"3a406a87-d961-4352-b049-eb5476b6f7a5\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-s6nf9" Sep 11 18:16:35.266023 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.265970 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvdkm\" (UniqueName: \"kubernetes.io/projected/3a406a87-d961-4352-b049-eb5476b6f7a5-kube-api-access-bvdkm\") pod \"cluster-monitoring-operator-6cd954b897-s6nf9\" (UID: \"3a406a87-d961-4352-b049-eb5476b6f7a5\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-s6nf9" Sep 11 18:16:35.281686 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.281658 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-s6nf9"] Sep 11 18:16:35.322233 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.322200 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-jfk78"] Sep 11 18:16:35.323789 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.323774 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-jfk78" Sep 11 18:16:35.325940 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.325914 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-2szfj"] Sep 11 18:16:35.326251 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.326221 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 11 18:16:35.326356 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.326253 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-d7rrj\"" Sep 11 18:16:35.326356 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.326223 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 11 18:16:35.327651 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.327634 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-dj524"] Sep 11 18:16:35.327769 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.327751 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2szfj" Sep 11 18:16:35.329254 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.329238 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-dj524" Sep 11 18:16:35.330534 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.330517 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 11 18:16:35.330647 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.330531 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-pw2xh\"" Sep 11 18:16:35.330839 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.330824 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 11 18:16:35.332093 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.331817 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 11 18:16:35.332093 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.331862 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 11 18:16:35.332093 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.331870 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-9cjtf\"" Sep 11 18:16:35.332093 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.331878 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 11 18:16:35.332093 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.331931 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 11 18:16:35.332364 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.332344 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 11 18:16:35.332487 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.332426 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 11 18:16:35.335826 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.335804 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-jfk78"] Sep 11 18:16:35.337551 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.337530 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 11 18:16:35.348282 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.348261 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-dj524"] Sep 11 18:16:35.348905 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.348888 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-2szfj"] Sep 11 18:16:35.367226 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.367199 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6bdab508-3741-4fe3-8f60-233f57470db8-config\") pod \"service-ca-operator-cf7c776b7-2szfj\" (UID: \"6bdab508-3741-4fe3-8f60-233f57470db8\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2szfj" Sep 11 18:16:35.367226 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.367227 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/60702ef3-98ef-463b-b877-3a00825353d8-serving-cert\") pod \"console-operator-696884fb9c-dj524\" (UID: \"60702ef3-98ef-463b-b877-3a00825353d8\") " pod="openshift-console-operator/console-operator-696884fb9c-dj524" Sep 11 18:16:35.367447 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.367260 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx46k\" (UniqueName: \"kubernetes.io/projected/56ef9bd1-1315-4693-93a1-9cf6dc6065a0-kube-api-access-lx46k\") pod \"volume-data-source-validator-86695b7cc7-jfk78\" (UID: \"56ef9bd1-1315-4693-93a1-9cf6dc6065a0\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-jfk78" Sep 11 18:16:35.367447 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.367280 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3a406a87-d961-4352-b049-eb5476b6f7a5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-s6nf9\" (UID: \"3a406a87-d961-4352-b049-eb5476b6f7a5\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-s6nf9" Sep 11 18:16:35.367447 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:16:35.367378 2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 11 18:16:35.367447 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.367423 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvdkm\" (UniqueName: \"kubernetes.io/projected/3a406a87-d961-4352-b049-eb5476b6f7a5-kube-api-access-bvdkm\") pod \"cluster-monitoring-operator-6cd954b897-s6nf9\" (UID: \"3a406a87-d961-4352-b049-eb5476b6f7a5\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-s6nf9" Sep 11 18:16:35.367447 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:16:35.367441 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a406a87-d961-4352-b049-eb5476b6f7a5-cluster-monitoring-operator-tls podName:3a406a87-d961-4352-b049-eb5476b6f7a5 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:35.867425041 +0000 UTC m=+77.364442785 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3a406a87-d961-4352-b049-eb5476b6f7a5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-s6nf9" (UID: "3a406a87-d961-4352-b049-eb5476b6f7a5") : secret "cluster-monitoring-operator-tls" not found Sep 11 18:16:35.367695 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.367466 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6bdab508-3741-4fe3-8f60-233f57470db8-serving-cert\") pod \"service-ca-operator-cf7c776b7-2szfj\" (UID: \"6bdab508-3741-4fe3-8f60-233f57470db8\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2szfj" Sep 11 18:16:35.367695 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.367487 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/60702ef3-98ef-463b-b877-3a00825353d8-trusted-ca\") pod \"console-operator-696884fb9c-dj524\" (UID: \"60702ef3-98ef-463b-b877-3a00825353d8\") " pod="openshift-console-operator/console-operator-696884fb9c-dj524" Sep 11 18:16:35.367695 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.367522 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5fg5\" (UniqueName: \"kubernetes.io/projected/6bdab508-3741-4fe3-8f60-233f57470db8-kube-api-access-p5fg5\") pod \"service-ca-operator-cf7c776b7-2szfj\" (UID: \"6bdab508-3741-4fe3-8f60-233f57470db8\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2szfj" Sep 11 18:16:35.367695 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.367538 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/60702ef3-98ef-463b-b877-3a00825353d8-config\") pod \"console-operator-696884fb9c-dj524\" (UID: \"60702ef3-98ef-463b-b877-3a00825353d8\") " pod="openshift-console-operator/console-operator-696884fb9c-dj524" Sep 11 18:16:35.367695 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.367584 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsrmq\" (UniqueName: \"kubernetes.io/projected/60702ef3-98ef-463b-b877-3a00825353d8-kube-api-access-hsrmq\") pod \"console-operator-696884fb9c-dj524\" (UID: \"60702ef3-98ef-463b-b877-3a00825353d8\") " pod="openshift-console-operator/console-operator-696884fb9c-dj524" Sep 11 18:16:35.367695 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.367613 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3a406a87-d961-4352-b049-eb5476b6f7a5-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-s6nf9\" (UID: \"3a406a87-d961-4352-b049-eb5476b6f7a5\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-s6nf9" Sep 11 18:16:35.368228 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.368212 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3a406a87-d961-4352-b049-eb5476b6f7a5-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-s6nf9\" (UID: \"3a406a87-d961-4352-b049-eb5476b6f7a5\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-s6nf9" Sep 11 18:16:35.377275 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.377252 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvdkm\" (UniqueName: \"kubernetes.io/projected/3a406a87-d961-4352-b049-eb5476b6f7a5-kube-api-access-bvdkm\") pod \"cluster-monitoring-operator-6cd954b897-s6nf9\" (UID: \"3a406a87-d961-4352-b049-eb5476b6f7a5\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-s6nf9" Sep 11 18:16:35.409963 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.409930 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9pjz5"] Sep 11 18:16:35.411638 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.411620 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9pjz5" Sep 11 18:16:35.413242 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.413210 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-spcnr"] Sep 11 18:16:35.414771 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.414752 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-spcnr" Sep 11 18:16:35.415195 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.415174 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 11 18:16:35.415290 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.415195 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 11 18:16:35.415290 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.415224 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-sd52v\"" Sep 11 18:16:35.415290 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.415182 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 11 18:16:35.415550 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.415531 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 11 18:16:35.416614 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.416597 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 11 18:16:35.417092 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.417075 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 11 18:16:35.417240 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.417221 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 11 18:16:35.417808 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.417790 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 11 18:16:35.417901 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.417796 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-cw66c\"" Sep 11 18:16:35.425154 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.425135 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 11 18:16:35.425959 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.425894 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9pjz5"] Sep 11 18:16:35.432989 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.432967 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-spcnr"] Sep 11 18:16:35.468431 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.468395 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsrmq\" (UniqueName: \"kubernetes.io/projected/60702ef3-98ef-463b-b877-3a00825353d8-kube-api-access-hsrmq\") pod \"console-operator-696884fb9c-dj524\" (UID: \"60702ef3-98ef-463b-b877-3a00825353d8\") " pod="openshift-console-operator/console-operator-696884fb9c-dj524" Sep 11 18:16:35.468624 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.468448 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/73a83e98-5fd7-43a6-85cc-ffd8d36057cf-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-9pjz5\" (UID: \"73a83e98-5fd7-43a6-85cc-ffd8d36057cf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9pjz5" Sep 11 18:16:35.468624 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.468469 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/97808c4d-b62a-4ca1-b686-5e27a9aa79a0-snapshots\") pod \"insights-operator-5844b856db-spcnr\" (UID: \"97808c4d-b62a-4ca1-b686-5e27a9aa79a0\") " pod="openshift-insights/insights-operator-5844b856db-spcnr" Sep 11 18:16:35.468624 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.468490 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6bdab508-3741-4fe3-8f60-233f57470db8-config\") pod \"service-ca-operator-cf7c776b7-2szfj\" (UID: \"6bdab508-3741-4fe3-8f60-233f57470db8\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2szfj" Sep 11 18:16:35.468777 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.468618 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/60702ef3-98ef-463b-b877-3a00825353d8-serving-cert\") pod \"console-operator-696884fb9c-dj524\" (UID: \"60702ef3-98ef-463b-b877-3a00825353d8\") " pod="openshift-console-operator/console-operator-696884fb9c-dj524" Sep 11 18:16:35.468777 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.468664 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/73a83e98-5fd7-43a6-85cc-ffd8d36057cf-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-9pjz5\" (UID: \"73a83e98-5fd7-43a6-85cc-ffd8d36057cf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9pjz5" Sep 11 18:16:35.468777 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.468693 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvcn7\" (UniqueName: \"kubernetes.io/projected/73a83e98-5fd7-43a6-85cc-ffd8d36057cf-kube-api-access-kvcn7\") pod \"kube-storage-version-migrator-operator-6c8f84f897-9pjz5\" (UID: \"73a83e98-5fd7-43a6-85cc-ffd8d36057cf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9pjz5" Sep 11 18:16:35.468777 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.468746 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97808c4d-b62a-4ca1-b686-5e27a9aa79a0-tmp\") pod \"insights-operator-5844b856db-spcnr\" (UID: \"97808c4d-b62a-4ca1-b686-5e27a9aa79a0\") " pod="openshift-insights/insights-operator-5844b856db-spcnr" Sep 11 18:16:35.468967 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.468783 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khq2v\" (UniqueName: \"kubernetes.io/projected/97808c4d-b62a-4ca1-b686-5e27a9aa79a0-kube-api-access-khq2v\") pod \"insights-operator-5844b856db-spcnr\" (UID: \"97808c4d-b62a-4ca1-b686-5e27a9aa79a0\") " pod="openshift-insights/insights-operator-5844b856db-spcnr" Sep 11 18:16:35.468967 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.468822 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx46k\" (UniqueName: \"kubernetes.io/projected/56ef9bd1-1315-4693-93a1-9cf6dc6065a0-kube-api-access-lx46k\") pod \"volume-data-source-validator-86695b7cc7-jfk78\" (UID: \"56ef9bd1-1315-4693-93a1-9cf6dc6065a0\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-jfk78" Sep 11 18:16:35.468967 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.468849 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97808c4d-b62a-4ca1-b686-5e27a9aa79a0-trusted-ca-bundle\") pod \"insights-operator-5844b856db-spcnr\" (UID: \"97808c4d-b62a-4ca1-b686-5e27a9aa79a0\") " pod="openshift-insights/insights-operator-5844b856db-spcnr" Sep 11 18:16:35.468967 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.468867 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97808c4d-b62a-4ca1-b686-5e27a9aa79a0-service-ca-bundle\") pod \"insights-operator-5844b856db-spcnr\" (UID: \"97808c4d-b62a-4ca1-b686-5e27a9aa79a0\") " pod="openshift-insights/insights-operator-5844b856db-spcnr" Sep 11 18:16:35.468967 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.468912 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6bdab508-3741-4fe3-8f60-233f57470db8-serving-cert\") pod \"service-ca-operator-cf7c776b7-2szfj\" (UID: \"6bdab508-3741-4fe3-8f60-233f57470db8\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2szfj" Sep 11 18:16:35.468967 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.468943 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/60702ef3-98ef-463b-b877-3a00825353d8-trusted-ca\") pod \"console-operator-696884fb9c-dj524\" (UID: \"60702ef3-98ef-463b-b877-3a00825353d8\") " pod="openshift-console-operator/console-operator-696884fb9c-dj524" Sep 11 18:16:35.469282 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.468974 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6bdab508-3741-4fe3-8f60-233f57470db8-config\") pod \"service-ca-operator-cf7c776b7-2szfj\" (UID: \"6bdab508-3741-4fe3-8f60-233f57470db8\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2szfj" Sep 11 18:16:35.469282 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.468995 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5fg5\" (UniqueName: \"kubernetes.io/projected/6bdab508-3741-4fe3-8f60-233f57470db8-kube-api-access-p5fg5\") pod \"service-ca-operator-cf7c776b7-2szfj\" (UID: \"6bdab508-3741-4fe3-8f60-233f57470db8\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2szfj" Sep 11 18:16:35.469282 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.469024 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/60702ef3-98ef-463b-b877-3a00825353d8-config\") pod \"console-operator-696884fb9c-dj524\" (UID: \"60702ef3-98ef-463b-b877-3a00825353d8\") " pod="openshift-console-operator/console-operator-696884fb9c-dj524" Sep 11 18:16:35.469282 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.469107 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/97808c4d-b62a-4ca1-b686-5e27a9aa79a0-serving-cert\") pod \"insights-operator-5844b856db-spcnr\" (UID: \"97808c4d-b62a-4ca1-b686-5e27a9aa79a0\") " pod="openshift-insights/insights-operator-5844b856db-spcnr" Sep 11 18:16:35.469656 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.469635 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/60702ef3-98ef-463b-b877-3a00825353d8-config\") pod \"console-operator-696884fb9c-dj524\" (UID: \"60702ef3-98ef-463b-b877-3a00825353d8\") " pod="openshift-console-operator/console-operator-696884fb9c-dj524" Sep 11 18:16:35.469863 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.469845 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/60702ef3-98ef-463b-b877-3a00825353d8-trusted-ca\") pod \"console-operator-696884fb9c-dj524\" (UID: \"60702ef3-98ef-463b-b877-3a00825353d8\") " pod="openshift-console-operator/console-operator-696884fb9c-dj524" Sep 11 18:16:35.471124 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.471091 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/60702ef3-98ef-463b-b877-3a00825353d8-serving-cert\") pod \"console-operator-696884fb9c-dj524\" (UID: \"60702ef3-98ef-463b-b877-3a00825353d8\") " pod="openshift-console-operator/console-operator-696884fb9c-dj524" Sep 11 18:16:35.471124 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.471119 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6bdab508-3741-4fe3-8f60-233f57470db8-serving-cert\") pod \"service-ca-operator-cf7c776b7-2szfj\" (UID: \"6bdab508-3741-4fe3-8f60-233f57470db8\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2szfj" Sep 11 18:16:35.480609 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.480552 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5fg5\" (UniqueName: \"kubernetes.io/projected/6bdab508-3741-4fe3-8f60-233f57470db8-kube-api-access-p5fg5\") pod \"service-ca-operator-cf7c776b7-2szfj\" (UID: \"6bdab508-3741-4fe3-8f60-233f57470db8\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2szfj" Sep 11 18:16:35.480851 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.480833 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx46k\" (UniqueName: \"kubernetes.io/projected/56ef9bd1-1315-4693-93a1-9cf6dc6065a0-kube-api-access-lx46k\") pod \"volume-data-source-validator-86695b7cc7-jfk78\" (UID: \"56ef9bd1-1315-4693-93a1-9cf6dc6065a0\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-jfk78" Sep 11 18:16:35.480928 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.480912 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsrmq\" (UniqueName: \"kubernetes.io/projected/60702ef3-98ef-463b-b877-3a00825353d8-kube-api-access-hsrmq\") pod \"console-operator-696884fb9c-dj524\" (UID: \"60702ef3-98ef-463b-b877-3a00825353d8\") " pod="openshift-console-operator/console-operator-696884fb9c-dj524" Sep 11 18:16:35.569586 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.569544 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97808c4d-b62a-4ca1-b686-5e27a9aa79a0-trusted-ca-bundle\") pod \"insights-operator-5844b856db-spcnr\" (UID: \"97808c4d-b62a-4ca1-b686-5e27a9aa79a0\") " pod="openshift-insights/insights-operator-5844b856db-spcnr" Sep 11 18:16:35.569586 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.569585 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97808c4d-b62a-4ca1-b686-5e27a9aa79a0-service-ca-bundle\") pod \"insights-operator-5844b856db-spcnr\" (UID: \"97808c4d-b62a-4ca1-b686-5e27a9aa79a0\") " pod="openshift-insights/insights-operator-5844b856db-spcnr" Sep 11 18:16:35.569778 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.569700 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/97808c4d-b62a-4ca1-b686-5e27a9aa79a0-serving-cert\") pod \"insights-operator-5844b856db-spcnr\" (UID: \"97808c4d-b62a-4ca1-b686-5e27a9aa79a0\") " pod="openshift-insights/insights-operator-5844b856db-spcnr" Sep 11 18:16:35.569778 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.569739 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/73a83e98-5fd7-43a6-85cc-ffd8d36057cf-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-9pjz5\" (UID: \"73a83e98-5fd7-43a6-85cc-ffd8d36057cf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9pjz5" Sep 11 18:16:35.569778 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.569755 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/97808c4d-b62a-4ca1-b686-5e27a9aa79a0-snapshots\") pod \"insights-operator-5844b856db-spcnr\" (UID: \"97808c4d-b62a-4ca1-b686-5e27a9aa79a0\") " pod="openshift-insights/insights-operator-5844b856db-spcnr" Sep 11 18:16:35.569778 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.569774 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/73a83e98-5fd7-43a6-85cc-ffd8d36057cf-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-9pjz5\" (UID: \"73a83e98-5fd7-43a6-85cc-ffd8d36057cf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9pjz5" Sep 11 18:16:35.569969 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.569845 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvcn7\" (UniqueName: \"kubernetes.io/projected/73a83e98-5fd7-43a6-85cc-ffd8d36057cf-kube-api-access-kvcn7\") pod \"kube-storage-version-migrator-operator-6c8f84f897-9pjz5\" (UID: \"73a83e98-5fd7-43a6-85cc-ffd8d36057cf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9pjz5" Sep 11 18:16:35.569969 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.569888 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97808c4d-b62a-4ca1-b686-5e27a9aa79a0-tmp\") pod \"insights-operator-5844b856db-spcnr\" (UID: \"97808c4d-b62a-4ca1-b686-5e27a9aa79a0\") " pod="openshift-insights/insights-operator-5844b856db-spcnr" Sep 11 18:16:35.569969 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.569916 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khq2v\" (UniqueName: \"kubernetes.io/projected/97808c4d-b62a-4ca1-b686-5e27a9aa79a0-kube-api-access-khq2v\") pod \"insights-operator-5844b856db-spcnr\" (UID: \"97808c4d-b62a-4ca1-b686-5e27a9aa79a0\") " pod="openshift-insights/insights-operator-5844b856db-spcnr" Sep 11 18:16:35.570399 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.570365 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97808c4d-b62a-4ca1-b686-5e27a9aa79a0-tmp\") pod \"insights-operator-5844b856db-spcnr\" (UID: \"97808c4d-b62a-4ca1-b686-5e27a9aa79a0\") " pod="openshift-insights/insights-operator-5844b856db-spcnr" Sep 11 18:16:35.570521 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.570363 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/73a83e98-5fd7-43a6-85cc-ffd8d36057cf-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-9pjz5\" (UID: \"73a83e98-5fd7-43a6-85cc-ffd8d36057cf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9pjz5" Sep 11 18:16:35.570585 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.570560 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97808c4d-b62a-4ca1-b686-5e27a9aa79a0-service-ca-bundle\") pod \"insights-operator-5844b856db-spcnr\" (UID: \"97808c4d-b62a-4ca1-b686-5e27a9aa79a0\") " pod="openshift-insights/insights-operator-5844b856db-spcnr" Sep 11 18:16:35.570642 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.570630 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97808c4d-b62a-4ca1-b686-5e27a9aa79a0-trusted-ca-bundle\") pod \"insights-operator-5844b856db-spcnr\" (UID: \"97808c4d-b62a-4ca1-b686-5e27a9aa79a0\") " pod="openshift-insights/insights-operator-5844b856db-spcnr" Sep 11 18:16:35.570744 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.570726 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/97808c4d-b62a-4ca1-b686-5e27a9aa79a0-snapshots\") pod \"insights-operator-5844b856db-spcnr\" (UID: \"97808c4d-b62a-4ca1-b686-5e27a9aa79a0\") " pod="openshift-insights/insights-operator-5844b856db-spcnr" Sep 11 18:16:35.572045 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.572014 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/73a83e98-5fd7-43a6-85cc-ffd8d36057cf-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-9pjz5\" (UID: \"73a83e98-5fd7-43a6-85cc-ffd8d36057cf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9pjz5" Sep 11 18:16:35.572184 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.572164 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/97808c4d-b62a-4ca1-b686-5e27a9aa79a0-serving-cert\") pod \"insights-operator-5844b856db-spcnr\" (UID: \"97808c4d-b62a-4ca1-b686-5e27a9aa79a0\") " pod="openshift-insights/insights-operator-5844b856db-spcnr" Sep 11 18:16:35.584563 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.584534 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvcn7\" (UniqueName: \"kubernetes.io/projected/73a83e98-5fd7-43a6-85cc-ffd8d36057cf-kube-api-access-kvcn7\") pod \"kube-storage-version-migrator-operator-6c8f84f897-9pjz5\" (UID: \"73a83e98-5fd7-43a6-85cc-ffd8d36057cf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9pjz5" Sep 11 18:16:35.584799 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.584783 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khq2v\" (UniqueName: \"kubernetes.io/projected/97808c4d-b62a-4ca1-b686-5e27a9aa79a0-kube-api-access-khq2v\") pod \"insights-operator-5844b856db-spcnr\" (UID: \"97808c4d-b62a-4ca1-b686-5e27a9aa79a0\") " pod="openshift-insights/insights-operator-5844b856db-spcnr" Sep 11 18:16:35.634576 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.634546 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-jfk78" Sep 11 18:16:35.642070 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.642044 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2szfj" Sep 11 18:16:35.645983 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.645961 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-dj524" Sep 11 18:16:35.721890 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.721826 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9pjz5" Sep 11 18:16:35.732101 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.732022 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-spcnr" Sep 11 18:16:35.872253 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.872206 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3a406a87-d961-4352-b049-eb5476b6f7a5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-s6nf9\" (UID: \"3a406a87-d961-4352-b049-eb5476b6f7a5\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-s6nf9" Sep 11 18:16:35.872426 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:16:35.872362 2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 11 18:16:35.872471 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:16:35.872427 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a406a87-d961-4352-b049-eb5476b6f7a5-cluster-monitoring-operator-tls podName:3a406a87-d961-4352-b049-eb5476b6f7a5 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:36.87241335 +0000 UTC m=+78.369431087 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3a406a87-d961-4352-b049-eb5476b6f7a5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-s6nf9" (UID: "3a406a87-d961-4352-b049-eb5476b6f7a5") : secret "cluster-monitoring-operator-tls" not found Sep 11 18:16:35.913910 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.913739 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-jfk78"] Sep 11 18:16:35.916647 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:16:35.916617 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56ef9bd1_1315_4693_93a1_9cf6dc6065a0.slice/crio-97b650b38123c0e6f75441168bbbc3ebd1c1cb279dc200a954a64694a7c6027d WatchSource:0}: Error finding container 97b650b38123c0e6f75441168bbbc3ebd1c1cb279dc200a954a64694a7c6027d: Status 404 returned error can't find the container with id 97b650b38123c0e6f75441168bbbc3ebd1c1cb279dc200a954a64694a7c6027d Sep 11 18:16:35.922138 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.922112 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-2szfj"] Sep 11 18:16:35.924963 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:16:35.924941 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bdab508_3741_4fe3_8f60_233f57470db8.slice/crio-863e80818e61c15d5c041f4e58609c3cd545678c74bc30c565897d4faa3a233e WatchSource:0}: Error finding container 863e80818e61c15d5c041f4e58609c3cd545678c74bc30c565897d4faa3a233e: Status 404 returned error can't find the container with id 863e80818e61c15d5c041f4e58609c3cd545678c74bc30c565897d4faa3a233e Sep 11 18:16:35.942813 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:35.942787 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-dj524"] Sep 11 18:16:35.945666 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:16:35.945640 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60702ef3_98ef_463b_b877_3a00825353d8.slice/crio-3876fd1232613f0de918f56ab25bd70a49309b0159147ad59de37ae1e08c1e52 WatchSource:0}: Error finding container 3876fd1232613f0de918f56ab25bd70a49309b0159147ad59de37ae1e08c1e52: Status 404 returned error can't find the container with id 3876fd1232613f0de918f56ab25bd70a49309b0159147ad59de37ae1e08c1e52 Sep 11 18:16:36.017965 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:36.017885 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9pjz5"] Sep 11 18:16:36.021157 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:36.021130 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-spcnr"] Sep 11 18:16:36.021441 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:16:36.021421 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73a83e98_5fd7_43a6_85cc_ffd8d36057cf.slice/crio-cf501014a1406665b064289a215f9c37f0af5292c6c23ea007c76b48f792858c WatchSource:0}: Error finding container cf501014a1406665b064289a215f9c37f0af5292c6c23ea007c76b48f792858c: Status 404 returned error can't find the container with id cf501014a1406665b064289a215f9c37f0af5292c6c23ea007c76b48f792858c Sep 11 18:16:36.024287 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:16:36.024266 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97808c4d_b62a_4ca1_b686_5e27a9aa79a0.slice/crio-9aca05570ff82d603948024d8cc6bde3f805bbe4283393e72fde14cf4dbb83d6 WatchSource:0}: Error finding container 9aca05570ff82d603948024d8cc6bde3f805bbe4283393e72fde14cf4dbb83d6: Status 404 returned error can't find the container with id 9aca05570ff82d603948024d8cc6bde3f805bbe4283393e72fde14cf4dbb83d6 Sep 11 18:16:36.431906 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:36.431867 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-spcnr" event={"ID":"97808c4d-b62a-4ca1-b686-5e27a9aa79a0","Type":"ContainerStarted","Data":"9aca05570ff82d603948024d8cc6bde3f805bbe4283393e72fde14cf4dbb83d6"} Sep 11 18:16:36.433256 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:36.433224 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9pjz5" event={"ID":"73a83e98-5fd7-43a6-85cc-ffd8d36057cf","Type":"ContainerStarted","Data":"cf501014a1406665b064289a215f9c37f0af5292c6c23ea007c76b48f792858c"} Sep 11 18:16:36.434785 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:36.434759 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2szfj" event={"ID":"6bdab508-3741-4fe3-8f60-233f57470db8","Type":"ContainerStarted","Data":"863e80818e61c15d5c041f4e58609c3cd545678c74bc30c565897d4faa3a233e"} Sep 11 18:16:36.436017 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:36.435995 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-jfk78" event={"ID":"56ef9bd1-1315-4693-93a1-9cf6dc6065a0","Type":"ContainerStarted","Data":"97b650b38123c0e6f75441168bbbc3ebd1c1cb279dc200a954a64694a7c6027d"} Sep 11 18:16:36.437278 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:36.437241 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-dj524" event={"ID":"60702ef3-98ef-463b-b877-3a00825353d8","Type":"ContainerStarted","Data":"3876fd1232613f0de918f56ab25bd70a49309b0159147ad59de37ae1e08c1e52"} Sep 11 18:16:36.881225 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:36.881103 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3a406a87-d961-4352-b049-eb5476b6f7a5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-s6nf9\" (UID: \"3a406a87-d961-4352-b049-eb5476b6f7a5\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-s6nf9" Sep 11 18:16:36.881396 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:16:36.881327 2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 11 18:16:36.881478 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:16:36.881408 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a406a87-d961-4352-b049-eb5476b6f7a5-cluster-monitoring-operator-tls podName:3a406a87-d961-4352-b049-eb5476b6f7a5 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:38.881385775 +0000 UTC m=+80.378403510 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3a406a87-d961-4352-b049-eb5476b6f7a5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-s6nf9" (UID: "3a406a87-d961-4352-b049-eb5476b6f7a5") : secret "cluster-monitoring-operator-tls" not found Sep 11 18:16:37.442420 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:37.442384 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-jfk78" event={"ID":"56ef9bd1-1315-4693-93a1-9cf6dc6065a0","Type":"ContainerStarted","Data":"e8fcc296c1c669f4810265ca69d0dfbffce20f45aacd3a0c8eb8bc285cc810b2"} Sep 11 18:16:37.460913 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:37.460853 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-jfk78" podStartSLOduration=1.446519547 podStartE2EDuration="2.460833376s" totalImagesPullingTime="1.014313829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:16:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:37.459150887 +0000 UTC m=+78.956168644" watchObservedRunningTime="2026-09-11 18:16:37.460833376 +0000 UTC m=+78.957851130" Sep 11 18:16:38.901325 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:38.901280 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3a406a87-d961-4352-b049-eb5476b6f7a5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-s6nf9\" (UID: \"3a406a87-d961-4352-b049-eb5476b6f7a5\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-s6nf9" Sep 11 18:16:38.901745 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:16:38.901443 2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 11 18:16:38.901745 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:16:38.901527 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a406a87-d961-4352-b049-eb5476b6f7a5-cluster-monitoring-operator-tls podName:3a406a87-d961-4352-b049-eb5476b6f7a5 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:42.901502634 +0000 UTC m=+84.398520390 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3a406a87-d961-4352-b049-eb5476b6f7a5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-s6nf9" (UID: "3a406a87-d961-4352-b049-eb5476b6f7a5") : secret "cluster-monitoring-operator-tls" not found Sep 11 18:16:40.454566 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:40.454534 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dj524_60702ef3-98ef-463b-b877-3a00825353d8/console-operator/0.log" Sep 11 18:16:40.455026 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:40.454581 2561 generic.go:356] "Generic (PLEG): container finished" podID="60702ef3-98ef-463b-b877-3a00825353d8" containerID="ba87441f3b215c67fd78aa0a0a6e581c39d9968ea067e7c582a6a2d1cbf6542c" exitCode=255 Sep 11 18:16:40.455026 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:40.454679 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-dj524" event={"ID":"60702ef3-98ef-463b-b877-3a00825353d8","Type":"ContainerDied","Data":"ba87441f3b215c67fd78aa0a0a6e581c39d9968ea067e7c582a6a2d1cbf6542c"} Sep 11 18:16:40.455026 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:40.454913 2561 scope.go:117] "RemoveContainer" containerID="ba87441f3b215c67fd78aa0a0a6e581c39d9968ea067e7c582a6a2d1cbf6542c" Sep 11 18:16:40.456666 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:40.456638 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-spcnr" event={"ID":"97808c4d-b62a-4ca1-b686-5e27a9aa79a0","Type":"ContainerStarted","Data":"08bcb5c73db4fa45a4933cd84c72c127b7df9ed99a1fd914f3dee49f652ce9ec"} Sep 11 18:16:40.458356 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:40.458333 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9pjz5" event={"ID":"73a83e98-5fd7-43a6-85cc-ffd8d36057cf","Type":"ContainerStarted","Data":"a482987bc35b6d92c0bdfddc2373c13edb8367c8d439b87345a317d74394d7b6"} Sep 11 18:16:40.460023 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:40.459985 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2szfj" event={"ID":"6bdab508-3741-4fe3-8f60-233f57470db8","Type":"ContainerStarted","Data":"4dbec84b86d46c7e582fa9ffbbc7b1b38c2fe18ec118359c35741c5d673f1050"} Sep 11 18:16:40.487690 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:40.487638 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9pjz5" podStartSLOduration=1.744073955 podStartE2EDuration="5.487617176s" totalImagesPullingTime="3.743543221s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:16:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:40.487365686 +0000 UTC m=+81.984383444" watchObservedRunningTime="2026-09-11 18:16:40.487617176 +0000 UTC m=+81.984634931" Sep 11 18:16:40.502659 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:40.502610 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2szfj" podStartSLOduration=1.657546432 podStartE2EDuration="5.502590289s" totalImagesPullingTime="3.845043857s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:16:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:40.50192074 +0000 UTC m=+81.998938497" watchObservedRunningTime="2026-09-11 18:16:40.502590289 +0000 UTC m=+81.999608044" Sep 11 18:16:40.990888 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:40.990787 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-spcnr" podStartSLOduration=2.24778918 podStartE2EDuration="5.990761684s" totalImagesPullingTime="3.742972504s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:16:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:40.519303321 +0000 UTC m=+82.016321077" watchObservedRunningTime="2026-09-11 18:16:40.990761684 +0000 UTC m=+82.487779440" Sep 11 18:16:40.991867 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:40.991845 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-c2vf2"] Sep 11 18:16:40.994293 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:40.994270 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-c2vf2" Sep 11 18:16:40.996511 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:40.996492 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 11 18:16:40.996741 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:40.996722 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 11 18:16:40.996741 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:40.996743 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fpnpf\"" Sep 11 18:16:41.007377 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:41.007351 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-c2vf2"] Sep 11 18:16:41.020253 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:41.020228 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b8a85f9a-991b-4751-925e-4e4a8a400a90-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c2vf2\" (UID: \"b8a85f9a-991b-4751-925e-4e4a8a400a90\") " pod="openshift-insights/insights-runtime-extractor-c2vf2" Sep 11 18:16:41.020378 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:41.020276 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b8a85f9a-991b-4751-925e-4e4a8a400a90-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c2vf2\" (UID: \"b8a85f9a-991b-4751-925e-4e4a8a400a90\") " pod="openshift-insights/insights-runtime-extractor-c2vf2" Sep 11 18:16:41.020378 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:41.020297 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q46x\" (UniqueName: \"kubernetes.io/projected/b8a85f9a-991b-4751-925e-4e4a8a400a90-kube-api-access-4q46x\") pod \"insights-runtime-extractor-c2vf2\" (UID: \"b8a85f9a-991b-4751-925e-4e4a8a400a90\") " pod="openshift-insights/insights-runtime-extractor-c2vf2" Sep 11 18:16:41.020378 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:41.020359 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b8a85f9a-991b-4751-925e-4e4a8a400a90-crio-socket\") pod \"insights-runtime-extractor-c2vf2\" (UID: \"b8a85f9a-991b-4751-925e-4e4a8a400a90\") " pod="openshift-insights/insights-runtime-extractor-c2vf2" Sep 11 18:16:41.020510 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:41.020393 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8a85f9a-991b-4751-925e-4e4a8a400a90-data-volume\") pod \"insights-runtime-extractor-c2vf2\" (UID: \"b8a85f9a-991b-4751-925e-4e4a8a400a90\") " pod="openshift-insights/insights-runtime-extractor-c2vf2" Sep 11 18:16:41.121211 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:41.121172 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b8a85f9a-991b-4751-925e-4e4a8a400a90-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c2vf2\" (UID: \"b8a85f9a-991b-4751-925e-4e4a8a400a90\") " pod="openshift-insights/insights-runtime-extractor-c2vf2" Sep 11 18:16:41.121211 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:41.121212 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q46x\" (UniqueName: \"kubernetes.io/projected/b8a85f9a-991b-4751-925e-4e4a8a400a90-kube-api-access-4q46x\") pod \"insights-runtime-extractor-c2vf2\" (UID: \"b8a85f9a-991b-4751-925e-4e4a8a400a90\") " pod="openshift-insights/insights-runtime-extractor-c2vf2" Sep 11 18:16:41.121433 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:41.121230 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b8a85f9a-991b-4751-925e-4e4a8a400a90-crio-socket\") pod \"insights-runtime-extractor-c2vf2\" (UID: \"b8a85f9a-991b-4751-925e-4e4a8a400a90\") " pod="openshift-insights/insights-runtime-extractor-c2vf2" Sep 11 18:16:41.121433 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:41.121314 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b8a85f9a-991b-4751-925e-4e4a8a400a90-crio-socket\") pod \"insights-runtime-extractor-c2vf2\" (UID: \"b8a85f9a-991b-4751-925e-4e4a8a400a90\") " pod="openshift-insights/insights-runtime-extractor-c2vf2" Sep 11 18:16:41.121433 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:16:41.121318 2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 18:16:41.121433 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:16:41.121379 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8a85f9a-991b-4751-925e-4e4a8a400a90-insights-runtime-extractor-tls podName:b8a85f9a-991b-4751-925e-4e4a8a400a90 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:41.621363971 +0000 UTC m=+83.118381707 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b8a85f9a-991b-4751-925e-4e4a8a400a90-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c2vf2" (UID: "b8a85f9a-991b-4751-925e-4e4a8a400a90") : secret "insights-runtime-extractor-tls" not found Sep 11 18:16:41.121433 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:41.121421 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8a85f9a-991b-4751-925e-4e4a8a400a90-data-volume\") pod \"insights-runtime-extractor-c2vf2\" (UID: \"b8a85f9a-991b-4751-925e-4e4a8a400a90\") " pod="openshift-insights/insights-runtime-extractor-c2vf2" Sep 11 18:16:41.121642 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:41.121510 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b8a85f9a-991b-4751-925e-4e4a8a400a90-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c2vf2\" (UID: \"b8a85f9a-991b-4751-925e-4e4a8a400a90\") " pod="openshift-insights/insights-runtime-extractor-c2vf2" Sep 11 18:16:41.121642 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:16:41.121628 2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 18:16:41.121738 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:16:41.121690 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b8a85f9a-991b-4751-925e-4e4a8a400a90-kube-rbac-proxy-cm podName:b8a85f9a-991b-4751-925e-4e4a8a400a90 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:41.621672473 +0000 UTC m=+83.118690222 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b8a85f9a-991b-4751-925e-4e4a8a400a90-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c2vf2" (UID: "b8a85f9a-991b-4751-925e-4e4a8a400a90") : configmap "kube-rbac-proxy" not found Sep 11 18:16:41.121871 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:41.121851 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8a85f9a-991b-4751-925e-4e4a8a400a90-data-volume\") pod \"insights-runtime-extractor-c2vf2\" (UID: \"b8a85f9a-991b-4751-925e-4e4a8a400a90\") " pod="openshift-insights/insights-runtime-extractor-c2vf2" Sep 11 18:16:41.130724 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:41.130697 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q46x\" (UniqueName: \"kubernetes.io/projected/b8a85f9a-991b-4751-925e-4e4a8a400a90-kube-api-access-4q46x\") pod \"insights-runtime-extractor-c2vf2\" (UID: \"b8a85f9a-991b-4751-925e-4e4a8a400a90\") " pod="openshift-insights/insights-runtime-extractor-c2vf2" Sep 11 18:16:41.465124 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:41.465093 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dj524_60702ef3-98ef-463b-b877-3a00825353d8/console-operator/1.log" Sep 11 18:16:41.465582 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:41.465564 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dj524_60702ef3-98ef-463b-b877-3a00825353d8/console-operator/0.log" Sep 11 18:16:41.465662 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:41.465610 2561 generic.go:356] "Generic (PLEG): container finished" podID="60702ef3-98ef-463b-b877-3a00825353d8" containerID="6f37c812b1bc718140125e449b668c9b652c5024ec59a57584610b61be8e1856" exitCode=255 Sep 11 18:16:41.465739 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:41.465716 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-dj524" event={"ID":"60702ef3-98ef-463b-b877-3a00825353d8","Type":"ContainerDied","Data":"6f37c812b1bc718140125e449b668c9b652c5024ec59a57584610b61be8e1856"} Sep 11 18:16:41.465815 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:41.465802 2561 scope.go:117] "RemoveContainer" containerID="ba87441f3b215c67fd78aa0a0a6e581c39d9968ea067e7c582a6a2d1cbf6542c" Sep 11 18:16:41.465925 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:41.465902 2561 scope.go:117] "RemoveContainer" containerID="6f37c812b1bc718140125e449b668c9b652c5024ec59a57584610b61be8e1856" Sep 11 18:16:41.466142 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:16:41.466121 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-dj524_openshift-console-operator(60702ef3-98ef-463b-b877-3a00825353d8)\"" pod="openshift-console-operator/console-operator-696884fb9c-dj524" podUID="60702ef3-98ef-463b-b877-3a00825353d8" Sep 11 18:16:41.627617 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:41.627576 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b8a85f9a-991b-4751-925e-4e4a8a400a90-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c2vf2\" (UID: \"b8a85f9a-991b-4751-925e-4e4a8a400a90\") " pod="openshift-insights/insights-runtime-extractor-c2vf2" Sep 11 18:16:41.627773 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:41.627635 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b8a85f9a-991b-4751-925e-4e4a8a400a90-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c2vf2\" (UID: \"b8a85f9a-991b-4751-925e-4e4a8a400a90\") " pod="openshift-insights/insights-runtime-extractor-c2vf2" Sep 11 18:16:41.627773 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:16:41.627717 2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 18:16:41.627901 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:16:41.627777 2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 18:16:41.627901 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:16:41.627791 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b8a85f9a-991b-4751-925e-4e4a8a400a90-kube-rbac-proxy-cm podName:b8a85f9a-991b-4751-925e-4e4a8a400a90 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:42.627771418 +0000 UTC m=+84.124789150 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b8a85f9a-991b-4751-925e-4e4a8a400a90-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c2vf2" (UID: "b8a85f9a-991b-4751-925e-4e4a8a400a90") : configmap "kube-rbac-proxy" not found Sep 11 18:16:41.627901 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:16:41.627825 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8a85f9a-991b-4751-925e-4e4a8a400a90-insights-runtime-extractor-tls podName:b8a85f9a-991b-4751-925e-4e4a8a400a90 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:42.627812962 +0000 UTC m=+84.124830694 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b8a85f9a-991b-4751-925e-4e4a8a400a90-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c2vf2" (UID: "b8a85f9a-991b-4751-925e-4e4a8a400a90") : secret "insights-runtime-extractor-tls" not found Sep 11 18:16:42.470332 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:42.470305 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dj524_60702ef3-98ef-463b-b877-3a00825353d8/console-operator/1.log" Sep 11 18:16:42.470775 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:42.470642 2561 scope.go:117] "RemoveContainer" containerID="6f37c812b1bc718140125e449b668c9b652c5024ec59a57584610b61be8e1856" Sep 11 18:16:42.470816 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:16:42.470792 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-dj524_openshift-console-operator(60702ef3-98ef-463b-b877-3a00825353d8)\"" pod="openshift-console-operator/console-operator-696884fb9c-dj524" podUID="60702ef3-98ef-463b-b877-3a00825353d8" Sep 11 18:16:42.636099 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:42.636033 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b8a85f9a-991b-4751-925e-4e4a8a400a90-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c2vf2\" (UID: \"b8a85f9a-991b-4751-925e-4e4a8a400a90\") " pod="openshift-insights/insights-runtime-extractor-c2vf2" Sep 11 18:16:42.636272 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:42.636153 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b8a85f9a-991b-4751-925e-4e4a8a400a90-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c2vf2\" (UID: \"b8a85f9a-991b-4751-925e-4e4a8a400a90\") " pod="openshift-insights/insights-runtime-extractor-c2vf2" Sep 11 18:16:42.636272 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:16:42.636205 2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 18:16:42.636272 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:16:42.636239 2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 18:16:42.636372 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:16:42.636276 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8a85f9a-991b-4751-925e-4e4a8a400a90-insights-runtime-extractor-tls podName:b8a85f9a-991b-4751-925e-4e4a8a400a90 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:44.636261517 +0000 UTC m=+86.133279248 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b8a85f9a-991b-4751-925e-4e4a8a400a90-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c2vf2" (UID: "b8a85f9a-991b-4751-925e-4e4a8a400a90") : secret "insights-runtime-extractor-tls" not found Sep 11 18:16:42.636372 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:16:42.636291 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b8a85f9a-991b-4751-925e-4e4a8a400a90-kube-rbac-proxy-cm podName:b8a85f9a-991b-4751-925e-4e4a8a400a90 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:44.636285287 +0000 UTC m=+86.133303018 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b8a85f9a-991b-4751-925e-4e4a8a400a90-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c2vf2" (UID: "b8a85f9a-991b-4751-925e-4e4a8a400a90") : configmap "kube-rbac-proxy" not found Sep 11 18:16:42.938378 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:42.938345 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3a406a87-d961-4352-b049-eb5476b6f7a5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-s6nf9\" (UID: \"3a406a87-d961-4352-b049-eb5476b6f7a5\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-s6nf9" Sep 11 18:16:42.938540 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:16:42.938503 2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 11 18:16:42.938588 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:16:42.938578 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a406a87-d961-4352-b049-eb5476b6f7a5-cluster-monitoring-operator-tls podName:3a406a87-d961-4352-b049-eb5476b6f7a5 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:50.938555464 +0000 UTC m=+92.435573205 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3a406a87-d961-4352-b049-eb5476b6f7a5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-s6nf9" (UID: "3a406a87-d961-4352-b049-eb5476b6f7a5") : secret "cluster-monitoring-operator-tls" not found Sep 11 18:16:43.149299 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:43.149270 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xh28n_7f7d5659-22fb-43f1-8970-47837d126795/dns-node-resolver/0.log" Sep 11 18:16:43.750489 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:43.750461 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-k7qjg_6547cc6d-cb31-474a-8065-7fd47766fe02/node-ca/0.log" Sep 11 18:16:44.027637 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:44.027559 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-xtrrx"] Sep 11 18:16:44.029883 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:44.029860 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-xtrrx" Sep 11 18:16:44.032358 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:44.032336 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Sep 11 18:16:44.032483 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:44.032336 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Sep 11 18:16:44.032483 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:44.032386 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Sep 11 18:16:44.033160 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:44.033141 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-q4b9x\"" Sep 11 18:16:44.033160 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:44.033154 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Sep 11 18:16:44.038547 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:44.038524 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-xtrrx"] Sep 11 18:16:44.146689 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:44.146648 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6741efa1-5045-4e3d-92bc-12c44971f8a2-signing-key\") pod \"service-ca-568869fdc5-xtrrx\" (UID: \"6741efa1-5045-4e3d-92bc-12c44971f8a2\") " pod="openshift-service-ca/service-ca-568869fdc5-xtrrx" Sep 11 18:16:44.146875 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:44.146707 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt6k7\" (UniqueName: \"kubernetes.io/projected/6741efa1-5045-4e3d-92bc-12c44971f8a2-kube-api-access-rt6k7\") pod \"service-ca-568869fdc5-xtrrx\" (UID: \"6741efa1-5045-4e3d-92bc-12c44971f8a2\") " pod="openshift-service-ca/service-ca-568869fdc5-xtrrx" Sep 11 18:16:44.146875 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:44.146740 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6741efa1-5045-4e3d-92bc-12c44971f8a2-signing-cabundle\") pod \"service-ca-568869fdc5-xtrrx\" (UID: \"6741efa1-5045-4e3d-92bc-12c44971f8a2\") " pod="openshift-service-ca/service-ca-568869fdc5-xtrrx" Sep 11 18:16:44.247908 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:44.247866 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6741efa1-5045-4e3d-92bc-12c44971f8a2-signing-key\") pod \"service-ca-568869fdc5-xtrrx\" (UID: \"6741efa1-5045-4e3d-92bc-12c44971f8a2\") " pod="openshift-service-ca/service-ca-568869fdc5-xtrrx" Sep 11 18:16:44.248144 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:44.247920 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt6k7\" (UniqueName: \"kubernetes.io/projected/6741efa1-5045-4e3d-92bc-12c44971f8a2-kube-api-access-rt6k7\") pod \"service-ca-568869fdc5-xtrrx\" (UID: \"6741efa1-5045-4e3d-92bc-12c44971f8a2\") " pod="openshift-service-ca/service-ca-568869fdc5-xtrrx" Sep 11 18:16:44.248144 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:44.247947 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6741efa1-5045-4e3d-92bc-12c44971f8a2-signing-cabundle\") pod \"service-ca-568869fdc5-xtrrx\" (UID: \"6741efa1-5045-4e3d-92bc-12c44971f8a2\") " pod="openshift-service-ca/service-ca-568869fdc5-xtrrx" Sep 11 18:16:44.248723 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:44.248701 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6741efa1-5045-4e3d-92bc-12c44971f8a2-signing-cabundle\") pod \"service-ca-568869fdc5-xtrrx\" (UID: \"6741efa1-5045-4e3d-92bc-12c44971f8a2\") " pod="openshift-service-ca/service-ca-568869fdc5-xtrrx" Sep 11 18:16:44.250340 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:44.250316 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6741efa1-5045-4e3d-92bc-12c44971f8a2-signing-key\") pod \"service-ca-568869fdc5-xtrrx\" (UID: \"6741efa1-5045-4e3d-92bc-12c44971f8a2\") " pod="openshift-service-ca/service-ca-568869fdc5-xtrrx" Sep 11 18:16:44.256969 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:44.256938 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt6k7\" (UniqueName: \"kubernetes.io/projected/6741efa1-5045-4e3d-92bc-12c44971f8a2-kube-api-access-rt6k7\") pod \"service-ca-568869fdc5-xtrrx\" (UID: \"6741efa1-5045-4e3d-92bc-12c44971f8a2\") " pod="openshift-service-ca/service-ca-568869fdc5-xtrrx" Sep 11 18:16:44.339673 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:44.339593 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-xtrrx" Sep 11 18:16:44.625948 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:44.625852 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-xtrrx"] Sep 11 18:16:44.630542 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:16:44.630519 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6741efa1_5045_4e3d_92bc_12c44971f8a2.slice/crio-cd28973f43aed8e094e71a50105f99b7a90df40f4846983aa22287abc19c8033 WatchSource:0}: Error finding container cd28973f43aed8e094e71a50105f99b7a90df40f4846983aa22287abc19c8033: Status 404 returned error can't find the container with id cd28973f43aed8e094e71a50105f99b7a90df40f4846983aa22287abc19c8033 Sep 11 18:16:44.650863 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:44.650838 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b8a85f9a-991b-4751-925e-4e4a8a400a90-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c2vf2\" (UID: \"b8a85f9a-991b-4751-925e-4e4a8a400a90\") " pod="openshift-insights/insights-runtime-extractor-c2vf2" Sep 11 18:16:44.651005 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:44.650886 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b8a85f9a-991b-4751-925e-4e4a8a400a90-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c2vf2\" (UID: \"b8a85f9a-991b-4751-925e-4e4a8a400a90\") " pod="openshift-insights/insights-runtime-extractor-c2vf2" Sep 11 18:16:44.651005 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:16:44.650978 2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 18:16:44.651145 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:16:44.650986 2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 18:16:44.651145 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:16:44.651048 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b8a85f9a-991b-4751-925e-4e4a8a400a90-kube-rbac-proxy-cm podName:b8a85f9a-991b-4751-925e-4e4a8a400a90 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:48.651027287 +0000 UTC m=+90.148045048 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b8a85f9a-991b-4751-925e-4e4a8a400a90-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c2vf2" (UID: "b8a85f9a-991b-4751-925e-4e4a8a400a90") : configmap "kube-rbac-proxy" not found Sep 11 18:16:44.651145 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:16:44.651101 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8a85f9a-991b-4751-925e-4e4a8a400a90-insights-runtime-extractor-tls podName:b8a85f9a-991b-4751-925e-4e4a8a400a90 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:48.651084277 +0000 UTC m=+90.148102035 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b8a85f9a-991b-4751-925e-4e4a8a400a90-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c2vf2" (UID: "b8a85f9a-991b-4751-925e-4e4a8a400a90") : secret "insights-runtime-extractor-tls" not found Sep 11 18:16:44.949798 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:44.949758 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-x6rnn_2c3feaeb-94a3-40e2-b158-976547caf818/serve-healthcheck-canary/0.log" Sep 11 18:16:45.483362 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:45.483326 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-xtrrx" event={"ID":"6741efa1-5045-4e3d-92bc-12c44971f8a2","Type":"ContainerStarted","Data":"92883d84c64e41b93dacdc64d73e5d2d6b3b53184dbf96b5fb9fa29af96c2abc"} Sep 11 18:16:45.483568 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:45.483374 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-xtrrx" event={"ID":"6741efa1-5045-4e3d-92bc-12c44971f8a2","Type":"ContainerStarted","Data":"cd28973f43aed8e094e71a50105f99b7a90df40f4846983aa22287abc19c8033"} Sep 11 18:16:45.500394 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:45.500343 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-xtrrx" podStartSLOduration=1.500325839 podStartE2EDuration="1.500325839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:16:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:45.499436355 +0000 UTC m=+86.996454137" watchObservedRunningTime="2026-09-11 18:16:45.500325839 +0000 UTC m=+86.997343592" Sep 11 18:16:45.646353 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:45.646306 2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-dj524" Sep 11 18:16:45.646548 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:45.646373 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-dj524" Sep 11 18:16:45.646733 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:45.646719 2561 scope.go:117] "RemoveContainer" containerID="6f37c812b1bc718140125e449b668c9b652c5024ec59a57584610b61be8e1856" Sep 11 18:16:45.646883 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:16:45.646866 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-dj524_openshift-console-operator(60702ef3-98ef-463b-b877-3a00825353d8)\"" pod="openshift-console-operator/console-operator-696884fb9c-dj524" podUID="60702ef3-98ef-463b-b877-3a00825353d8" Sep 11 18:16:48.684770 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:48.684732 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b8a85f9a-991b-4751-925e-4e4a8a400a90-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c2vf2\" (UID: \"b8a85f9a-991b-4751-925e-4e4a8a400a90\") " pod="openshift-insights/insights-runtime-extractor-c2vf2" Sep 11 18:16:48.685186 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:48.684841 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b8a85f9a-991b-4751-925e-4e4a8a400a90-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c2vf2\" (UID: \"b8a85f9a-991b-4751-925e-4e4a8a400a90\") " pod="openshift-insights/insights-runtime-extractor-c2vf2" Sep 11 18:16:48.685186 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:16:48.684891 2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 18:16:48.685186 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:16:48.684922 2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 18:16:48.685186 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:16:48.684964 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8a85f9a-991b-4751-925e-4e4a8a400a90-insights-runtime-extractor-tls podName:b8a85f9a-991b-4751-925e-4e4a8a400a90 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:56.684945544 +0000 UTC m=+98.181963290 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b8a85f9a-991b-4751-925e-4e4a8a400a90-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c2vf2" (UID: "b8a85f9a-991b-4751-925e-4e4a8a400a90") : secret "insights-runtime-extractor-tls" not found Sep 11 18:16:48.685186 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:16:48.684979 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b8a85f9a-991b-4751-925e-4e4a8a400a90-kube-rbac-proxy-cm podName:b8a85f9a-991b-4751-925e-4e4a8a400a90 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:56.684972713 +0000 UTC m=+98.181990446 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b8a85f9a-991b-4751-925e-4e4a8a400a90-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c2vf2" (UID: "b8a85f9a-991b-4751-925e-4e4a8a400a90") : configmap "kube-rbac-proxy" not found Sep 11 18:16:51.007283 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:51.007234 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3a406a87-d961-4352-b049-eb5476b6f7a5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-s6nf9\" (UID: \"3a406a87-d961-4352-b049-eb5476b6f7a5\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-s6nf9" Sep 11 18:16:51.007777 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:16:51.007417 2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 11 18:16:51.007777 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:16:51.007510 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a406a87-d961-4352-b049-eb5476b6f7a5-cluster-monitoring-operator-tls podName:3a406a87-d961-4352-b049-eb5476b6f7a5 nodeName:}" failed. No retries permitted until 2026-09-11 18:17:07.007486519 +0000 UTC m=+108.504504273 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3a406a87-d961-4352-b049-eb5476b6f7a5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-s6nf9" (UID: "3a406a87-d961-4352-b049-eb5476b6f7a5") : secret "cluster-monitoring-operator-tls" not found Sep 11 18:16:56.758038 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:56.757998 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b8a85f9a-991b-4751-925e-4e4a8a400a90-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c2vf2\" (UID: \"b8a85f9a-991b-4751-925e-4e4a8a400a90\") " pod="openshift-insights/insights-runtime-extractor-c2vf2" Sep 11 18:16:56.758483 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:56.758081 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b8a85f9a-991b-4751-925e-4e4a8a400a90-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c2vf2\" (UID: \"b8a85f9a-991b-4751-925e-4e4a8a400a90\") " pod="openshift-insights/insights-runtime-extractor-c2vf2" Sep 11 18:16:56.758609 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:56.758589 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b8a85f9a-991b-4751-925e-4e4a8a400a90-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c2vf2\" (UID: \"b8a85f9a-991b-4751-925e-4e4a8a400a90\") " pod="openshift-insights/insights-runtime-extractor-c2vf2" Sep 11 18:16:56.760424 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:56.760405 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b8a85f9a-991b-4751-925e-4e4a8a400a90-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c2vf2\" (UID: \"b8a85f9a-991b-4751-925e-4e4a8a400a90\") " pod="openshift-insights/insights-runtime-extractor-c2vf2" Sep 11 18:16:56.906349 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:56.906307 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-c2vf2" Sep 11 18:16:57.182851 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:57.182796 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-c2vf2"] Sep 11 18:16:57.186427 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:16:57.186392 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8a85f9a_991b_4751_925e_4e4a8a400a90.slice/crio-7b9d2448455d8b5756cb3292551c67e0186fab4ea8dace2ed073715b31465d95 WatchSource:0}: Error finding container 7b9d2448455d8b5756cb3292551c67e0186fab4ea8dace2ed073715b31465d95: Status 404 returned error can't find the container with id 7b9d2448455d8b5756cb3292551c67e0186fab4ea8dace2ed073715b31465d95 Sep 11 18:16:57.525828 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:57.525739 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c2vf2" event={"ID":"b8a85f9a-991b-4751-925e-4e4a8a400a90","Type":"ContainerStarted","Data":"629f8f31623d04afa0682b546cc4de51ce41246cf25a4da69648e0231839945f"} Sep 11 18:16:57.525828 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:57.525801 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c2vf2" event={"ID":"b8a85f9a-991b-4751-925e-4e4a8a400a90","Type":"ContainerStarted","Data":"7b9d2448455d8b5756cb3292551c67e0186fab4ea8dace2ed073715b31465d95"} Sep 11 18:16:58.531183 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:58.531097 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c2vf2" event={"ID":"b8a85f9a-991b-4751-925e-4e4a8a400a90","Type":"ContainerStarted","Data":"5e6b56bf144315f111440c7224afad9a9e6d2b6fa685078d738f0ed2ff68a148"} Sep 11 18:16:59.176646 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:59.176604 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71d0b928-d82f-4a8d-844c-36613596b07b-metrics-tls\") pod \"dns-default-c6nk8\" (UID: \"71d0b928-d82f-4a8d-844c-36613596b07b\") " pod="openshift-dns/dns-default-c6nk8" Sep 11 18:16:59.179536 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:59.179506 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71d0b928-d82f-4a8d-844c-36613596b07b-metrics-tls\") pod \"dns-default-c6nk8\" (UID: \"71d0b928-d82f-4a8d-844c-36613596b07b\") " pod="openshift-dns/dns-default-c6nk8" Sep 11 18:16:59.212307 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:59.212267 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-b9ltz\"" Sep 11 18:16:59.220165 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:59.220133 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c6nk8" Sep 11 18:16:59.414933 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:59.414903 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zr7xc" Sep 11 18:16:59.506899 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:16:59.506775 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-c6nk8"] Sep 11 18:16:59.893974 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:16:59.893892 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71d0b928_d82f_4a8d_844c_36613596b07b.slice/crio-b1f0a33436bf28a5be19cba2ff1b6ddfade3443adc84620c0120d483d7cfd908 WatchSource:0}: Error finding container b1f0a33436bf28a5be19cba2ff1b6ddfade3443adc84620c0120d483d7cfd908: Status 404 returned error can't find the container with id b1f0a33436bf28a5be19cba2ff1b6ddfade3443adc84620c0120d483d7cfd908 Sep 11 18:17:00.133459 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:00.133427 2561 scope.go:117] "RemoveContainer" containerID="6f37c812b1bc718140125e449b668c9b652c5024ec59a57584610b61be8e1856" Sep 11 18:17:00.545610 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:00.545529 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dj524_60702ef3-98ef-463b-b877-3a00825353d8/console-operator/1.log" Sep 11 18:17:00.545780 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:00.545632 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-dj524" event={"ID":"60702ef3-98ef-463b-b877-3a00825353d8","Type":"ContainerStarted","Data":"ac0f45753a6a6b2080e55d57159d4df99f0ebc7fffd22df3c9aff2d1c807b382"} Sep 11 18:17:00.546082 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:00.546009 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-dj524" Sep 11 18:17:00.547122 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:00.547094 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c6nk8" event={"ID":"71d0b928-d82f-4a8d-844c-36613596b07b","Type":"ContainerStarted","Data":"b1f0a33436bf28a5be19cba2ff1b6ddfade3443adc84620c0120d483d7cfd908"} Sep 11 18:17:00.549856 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:00.549829 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c2vf2" event={"ID":"b8a85f9a-991b-4751-925e-4e4a8a400a90","Type":"ContainerStarted","Data":"c6d95a543b723c8178be31867a95f04f6a6cc25c82f0f661ecca9557be019d58"} Sep 11 18:17:00.563983 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:00.563919 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-dj524" podStartSLOduration=21.739922882 podStartE2EDuration="25.563896246s" totalImagesPullingTime="3.823973364s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:16:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:17:00.563065484 +0000 UTC m=+102.060083232" watchObservedRunningTime="2026-09-11 18:17:00.563896246 +0000 UTC m=+102.060914001" Sep 11 18:17:00.584120 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:00.584038 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-c2vf2" podStartSLOduration=17.635261928 podStartE2EDuration="20.584009739s" totalImagesPullingTime="2.948747811s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:16:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:17:00.582613134 +0000 UTC m=+102.079630890" watchObservedRunningTime="2026-09-11 18:17:00.584009739 +0000 UTC m=+102.081027494" Sep 11 18:17:01.238074 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:01.238028 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-dj524" Sep 11 18:17:02.561049 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:02.560999 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c6nk8" event={"ID":"71d0b928-d82f-4a8d-844c-36613596b07b","Type":"ContainerStarted","Data":"3b4c522f1e12b702ece04f8175d02471c8c6cde33e8963e3c44a8a6040409a86"} Sep 11 18:17:02.561537 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:02.561104 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c6nk8" event={"ID":"71d0b928-d82f-4a8d-844c-36613596b07b","Type":"ContainerStarted","Data":"e10f55f60640ed6f4aa529857720d7e263b36a179b951521887e4f063d6d68bb"} Sep 11 18:17:02.561537 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:02.561382 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-c6nk8" Sep 11 18:17:02.587600 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:02.587494 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-c6nk8" podStartSLOduration=65.410735104 podStartE2EDuration="1m7.587473392s" totalImagesPullingTime="2.176738288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:15:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:17:02.587451322 +0000 UTC m=+104.084469076" watchObservedRunningTime="2026-09-11 18:17:02.587473392 +0000 UTC m=+104.084491147" Sep 11 18:17:02.595066 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:02.595031 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7454b9f8bb-7fkmr"] Sep 11 18:17:02.616375 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:02.616348 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7454b9f8bb-7fkmr"] Sep 11 18:17:02.616546 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:02.616450 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7454b9f8bb-7fkmr" Sep 11 18:17:02.618767 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:02.618739 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-8w6k7\"" Sep 11 18:17:02.619111 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:02.619085 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 11 18:17:02.619247 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:02.619120 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 11 18:17:02.619247 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:02.619216 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Sep 11 18:17:02.619708 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:02.619310 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 11 18:17:02.641466 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:02.641436 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6cd6c4f7f7-vh7q7"] Sep 11 18:17:02.660406 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:02.660371 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6cd6c4f7f7-vh7q7"] Sep 11 18:17:02.660592 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:02.660506 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6cd6c4f7f7-vh7q7" Sep 11 18:17:02.663087 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:02.663028 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 11 18:17:02.663364 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:02.663339 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 11 18:17:02.664375 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:02.664357 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vxws4\"" Sep 11 18:17:02.664482 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:02.664365 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 11 18:17:02.669665 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:02.669646 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 11 18:17:02.708554 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:02.708510 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp49w\" (UniqueName: \"kubernetes.io/projected/bedee96f-0acd-4d6b-9507-b14a79539821-kube-api-access-gp49w\") pod \"managed-serviceaccount-addon-agent-7454b9f8bb-7fkmr\" (UID: \"bedee96f-0acd-4d6b-9507-b14a79539821\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7454b9f8bb-7fkmr" Sep 11 18:17:02.708755 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:02.708714 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bedee96f-0acd-4d6b-9507-b14a79539821-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7454b9f8bb-7fkmr\" (UID: \"bedee96f-0acd-4d6b-9507-b14a79539821\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7454b9f8bb-7fkmr" Sep 11 18:17:02.809574 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:02.809541 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c6f21f03-eca4-45e5-80c1-63dee730248d-ca-trust-extracted\") pod \"image-registry-6cd6c4f7f7-vh7q7\" (UID: \"c6f21f03-eca4-45e5-80c1-63dee730248d\") " pod="openshift-image-registry/image-registry-6cd6c4f7f7-vh7q7" Sep 11 18:17:02.809574 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:02.809582 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c6f21f03-eca4-45e5-80c1-63dee730248d-image-registry-private-configuration\") pod \"image-registry-6cd6c4f7f7-vh7q7\" (UID: \"c6f21f03-eca4-45e5-80c1-63dee730248d\") " pod="openshift-image-registry/image-registry-6cd6c4f7f7-vh7q7" Sep 11 18:17:02.809843 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:02.809620 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bedee96f-0acd-4d6b-9507-b14a79539821-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7454b9f8bb-7fkmr\" (UID: \"bedee96f-0acd-4d6b-9507-b14a79539821\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7454b9f8bb-7fkmr" Sep 11 18:17:02.809843 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:02.809646 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c6f21f03-eca4-45e5-80c1-63dee730248d-registry-tls\") pod \"image-registry-6cd6c4f7f7-vh7q7\" (UID: \"c6f21f03-eca4-45e5-80c1-63dee730248d\") " pod="openshift-image-registry/image-registry-6cd6c4f7f7-vh7q7" Sep 11 18:17:02.809843 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:02.809668 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c6f21f03-eca4-45e5-80c1-63dee730248d-bound-sa-token\") pod \"image-registry-6cd6c4f7f7-vh7q7\" (UID: \"c6f21f03-eca4-45e5-80c1-63dee730248d\") " pod="openshift-image-registry/image-registry-6cd6c4f7f7-vh7q7" Sep 11 18:17:02.809843 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:02.809705 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf92c\" (UniqueName: \"kubernetes.io/projected/c6f21f03-eca4-45e5-80c1-63dee730248d-kube-api-access-bf92c\") pod \"image-registry-6cd6c4f7f7-vh7q7\" (UID: \"c6f21f03-eca4-45e5-80c1-63dee730248d\") " pod="openshift-image-registry/image-registry-6cd6c4f7f7-vh7q7" Sep 11 18:17:02.809843 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:02.809785 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c6f21f03-eca4-45e5-80c1-63dee730248d-registry-certificates\") pod \"image-registry-6cd6c4f7f7-vh7q7\" (UID: \"c6f21f03-eca4-45e5-80c1-63dee730248d\") " pod="openshift-image-registry/image-registry-6cd6c4f7f7-vh7q7" Sep 11 18:17:02.809843 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:02.809827 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c6f21f03-eca4-45e5-80c1-63dee730248d-installation-pull-secrets\") pod \"image-registry-6cd6c4f7f7-vh7q7\" (UID: \"c6f21f03-eca4-45e5-80c1-63dee730248d\") " pod="openshift-image-registry/image-registry-6cd6c4f7f7-vh7q7" Sep 11 18:17:02.810179 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:02.809885 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp49w\" (UniqueName: \"kubernetes.io/projected/bedee96f-0acd-4d6b-9507-b14a79539821-kube-api-access-gp49w\") pod \"managed-serviceaccount-addon-agent-7454b9f8bb-7fkmr\" (UID: \"bedee96f-0acd-4d6b-9507-b14a79539821\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7454b9f8bb-7fkmr" Sep 11 18:17:02.810179 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:02.809920 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c6f21f03-eca4-45e5-80c1-63dee730248d-trusted-ca\") pod \"image-registry-6cd6c4f7f7-vh7q7\" (UID: \"c6f21f03-eca4-45e5-80c1-63dee730248d\") " pod="openshift-image-registry/image-registry-6cd6c4f7f7-vh7q7" Sep 11 18:17:02.812249 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:02.812229 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bedee96f-0acd-4d6b-9507-b14a79539821-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7454b9f8bb-7fkmr\" (UID: \"bedee96f-0acd-4d6b-9507-b14a79539821\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7454b9f8bb-7fkmr" Sep 11 18:17:02.821375 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:02.821344 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp49w\" (UniqueName: \"kubernetes.io/projected/bedee96f-0acd-4d6b-9507-b14a79539821-kube-api-access-gp49w\") pod \"managed-serviceaccount-addon-agent-7454b9f8bb-7fkmr\" (UID: \"bedee96f-0acd-4d6b-9507-b14a79539821\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7454b9f8bb-7fkmr" Sep 11 18:17:02.910984 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:02.910950 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c6f21f03-eca4-45e5-80c1-63dee730248d-trusted-ca\") pod \"image-registry-6cd6c4f7f7-vh7q7\" (UID: \"c6f21f03-eca4-45e5-80c1-63dee730248d\") " pod="openshift-image-registry/image-registry-6cd6c4f7f7-vh7q7" Sep 11 18:17:02.911203 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:02.911006 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c6f21f03-eca4-45e5-80c1-63dee730248d-ca-trust-extracted\") pod \"image-registry-6cd6c4f7f7-vh7q7\" (UID: \"c6f21f03-eca4-45e5-80c1-63dee730248d\") " pod="openshift-image-registry/image-registry-6cd6c4f7f7-vh7q7" Sep 11 18:17:02.911203 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:02.911029 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c6f21f03-eca4-45e5-80c1-63dee730248d-image-registry-private-configuration\") pod \"image-registry-6cd6c4f7f7-vh7q7\" (UID: \"c6f21f03-eca4-45e5-80c1-63dee730248d\") " pod="openshift-image-registry/image-registry-6cd6c4f7f7-vh7q7" Sep 11 18:17:02.911203 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:02.911085 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c6f21f03-eca4-45e5-80c1-63dee730248d-registry-tls\") pod \"image-registry-6cd6c4f7f7-vh7q7\" (UID: \"c6f21f03-eca4-45e5-80c1-63dee730248d\") " pod="openshift-image-registry/image-registry-6cd6c4f7f7-vh7q7" Sep 11 18:17:02.911203 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:02.911103 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c6f21f03-eca4-45e5-80c1-63dee730248d-bound-sa-token\") pod \"image-registry-6cd6c4f7f7-vh7q7\" (UID: \"c6f21f03-eca4-45e5-80c1-63dee730248d\") " pod="openshift-image-registry/image-registry-6cd6c4f7f7-vh7q7" Sep 11 18:17:02.911656 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:02.911157 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf92c\" (UniqueName: \"kubernetes.io/projected/c6f21f03-eca4-45e5-80c1-63dee730248d-kube-api-access-bf92c\") pod \"image-registry-6cd6c4f7f7-vh7q7\" (UID: \"c6f21f03-eca4-45e5-80c1-63dee730248d\") " pod="openshift-image-registry/image-registry-6cd6c4f7f7-vh7q7" Sep 11 18:17:02.912208 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:02.911696 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c6f21f03-eca4-45e5-80c1-63dee730248d-registry-certificates\") pod \"image-registry-6cd6c4f7f7-vh7q7\" (UID: \"c6f21f03-eca4-45e5-80c1-63dee730248d\") " pod="openshift-image-registry/image-registry-6cd6c4f7f7-vh7q7" Sep 11 18:17:02.912208 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:02.911744 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c6f21f03-eca4-45e5-80c1-63dee730248d-installation-pull-secrets\") pod \"image-registry-6cd6c4f7f7-vh7q7\" (UID: \"c6f21f03-eca4-45e5-80c1-63dee730248d\") " pod="openshift-image-registry/image-registry-6cd6c4f7f7-vh7q7" Sep 11 18:17:02.912208 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:02.911858 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c6f21f03-eca4-45e5-80c1-63dee730248d-ca-trust-extracted\") pod \"image-registry-6cd6c4f7f7-vh7q7\" (UID: \"c6f21f03-eca4-45e5-80c1-63dee730248d\") " pod="openshift-image-registry/image-registry-6cd6c4f7f7-vh7q7" Sep 11 18:17:02.913120 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:02.913094 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c6f21f03-eca4-45e5-80c1-63dee730248d-trusted-ca\") pod \"image-registry-6cd6c4f7f7-vh7q7\" (UID: \"c6f21f03-eca4-45e5-80c1-63dee730248d\") " pod="openshift-image-registry/image-registry-6cd6c4f7f7-vh7q7" Sep 11 18:17:02.916076 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:02.914629 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c6f21f03-eca4-45e5-80c1-63dee730248d-image-registry-private-configuration\") pod \"image-registry-6cd6c4f7f7-vh7q7\" (UID: \"c6f21f03-eca4-45e5-80c1-63dee730248d\") " pod="openshift-image-registry/image-registry-6cd6c4f7f7-vh7q7" Sep 11 18:17:02.916076 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:02.914805 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c6f21f03-eca4-45e5-80c1-63dee730248d-installation-pull-secrets\") pod \"image-registry-6cd6c4f7f7-vh7q7\" (UID: \"c6f21f03-eca4-45e5-80c1-63dee730248d\") " pod="openshift-image-registry/image-registry-6cd6c4f7f7-vh7q7" Sep 11 18:17:02.916076 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:02.915241 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c6f21f03-eca4-45e5-80c1-63dee730248d-registry-certificates\") pod \"image-registry-6cd6c4f7f7-vh7q7\" (UID: \"c6f21f03-eca4-45e5-80c1-63dee730248d\") " pod="openshift-image-registry/image-registry-6cd6c4f7f7-vh7q7" Sep 11 18:17:02.917715 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:02.917693 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c6f21f03-eca4-45e5-80c1-63dee730248d-registry-tls\") pod \"image-registry-6cd6c4f7f7-vh7q7\" (UID: \"c6f21f03-eca4-45e5-80c1-63dee730248d\") " pod="openshift-image-registry/image-registry-6cd6c4f7f7-vh7q7" Sep 11 18:17:02.920420 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:02.920397 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c6f21f03-eca4-45e5-80c1-63dee730248d-bound-sa-token\") pod \"image-registry-6cd6c4f7f7-vh7q7\" (UID: \"c6f21f03-eca4-45e5-80c1-63dee730248d\") " pod="openshift-image-registry/image-registry-6cd6c4f7f7-vh7q7" Sep 11 18:17:02.924659 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:02.924638 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf92c\" (UniqueName: \"kubernetes.io/projected/c6f21f03-eca4-45e5-80c1-63dee730248d-kube-api-access-bf92c\") pod \"image-registry-6cd6c4f7f7-vh7q7\" (UID: \"c6f21f03-eca4-45e5-80c1-63dee730248d\") " pod="openshift-image-registry/image-registry-6cd6c4f7f7-vh7q7" Sep 11 18:17:02.935519 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:02.935495 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7454b9f8bb-7fkmr" Sep 11 18:17:02.972347 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:02.971972 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6cd6c4f7f7-vh7q7" Sep 11 18:17:03.218667 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:03.218298 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7454b9f8bb-7fkmr"] Sep 11 18:17:03.221269 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:17:03.221235 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbedee96f_0acd_4d6b_9507_b14a79539821.slice/crio-1bf8f5fe3452b7592a84df000285bf4816fe7053f99e4670fec2403916deea49 WatchSource:0}: Error finding container 1bf8f5fe3452b7592a84df000285bf4816fe7053f99e4670fec2403916deea49: Status 404 returned error can't find the container with id 1bf8f5fe3452b7592a84df000285bf4816fe7053f99e4670fec2403916deea49 Sep 11 18:17:03.250668 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:03.250644 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6cd6c4f7f7-vh7q7"] Sep 11 18:17:03.253105 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:17:03.253079 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6f21f03_eca4_45e5_80c1_63dee730248d.slice/crio-bafeec8991ab6bb615bf3fe07e94abe53d37a562c62185be74e1fb3ad38f7e00 WatchSource:0}: Error finding container bafeec8991ab6bb615bf3fe07e94abe53d37a562c62185be74e1fb3ad38f7e00: Status 404 returned error can't find the container with id bafeec8991ab6bb615bf3fe07e94abe53d37a562c62185be74e1fb3ad38f7e00 Sep 11 18:17:03.567967 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:03.567865 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6cd6c4f7f7-vh7q7" event={"ID":"c6f21f03-eca4-45e5-80c1-63dee730248d","Type":"ContainerStarted","Data":"ffad618313117c9135aec4c8a14463135adb5b7d188f7f4d5ad3fb562767b567"} Sep 11 18:17:03.567967 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:03.567917 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6cd6c4f7f7-vh7q7" event={"ID":"c6f21f03-eca4-45e5-80c1-63dee730248d","Type":"ContainerStarted","Data":"bafeec8991ab6bb615bf3fe07e94abe53d37a562c62185be74e1fb3ad38f7e00"} Sep 11 18:17:03.568506 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:03.567976 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6cd6c4f7f7-vh7q7" Sep 11 18:17:03.569462 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:03.569428 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7454b9f8bb-7fkmr" event={"ID":"bedee96f-0acd-4d6b-9507-b14a79539821","Type":"ContainerStarted","Data":"1bf8f5fe3452b7592a84df000285bf4816fe7053f99e4670fec2403916deea49"} Sep 11 18:17:03.591774 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:03.591707 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6cd6c4f7f7-vh7q7" podStartSLOduration=1.5916814270000001 podStartE2EDuration="1.591681427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:17:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:17:03.589759733 +0000 UTC m=+105.086777486" watchObservedRunningTime="2026-09-11 18:17:03.591681427 +0000 UTC m=+105.088699180" Sep 11 18:17:06.585799 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:06.585764 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7454b9f8bb-7fkmr" event={"ID":"bedee96f-0acd-4d6b-9507-b14a79539821","Type":"ContainerStarted","Data":"4a383bc6827de5d3669a0a4876cfde14dadb3eed3cac85aabc7e6a303ba9dee4"} Sep 11 18:17:06.609495 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:06.609443 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7454b9f8bb-7fkmr" podStartSLOduration=1.972841754 podStartE2EDuration="4.609424402s" totalImagesPullingTime="2.636582648s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:17:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:17:06.608607348 +0000 UTC m=+108.105625104" watchObservedRunningTime="2026-09-11 18:17:06.609424402 +0000 UTC m=+108.106442155" Sep 11 18:17:07.046391 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:07.046345 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3a406a87-d961-4352-b049-eb5476b6f7a5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-s6nf9\" (UID: \"3a406a87-d961-4352-b049-eb5476b6f7a5\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-s6nf9" Sep 11 18:17:07.048863 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:07.048829 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3a406a87-d961-4352-b049-eb5476b6f7a5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-s6nf9\" (UID: \"3a406a87-d961-4352-b049-eb5476b6f7a5\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-s6nf9" Sep 11 18:17:07.338410 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:07.338321 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-s6nf9" Sep 11 18:17:07.610250 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:07.610009 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-s6nf9"] Sep 11 18:17:07.612876 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:17:07.612850 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a406a87_d961_4352_b049_eb5476b6f7a5.slice/crio-b9be2e13878c18f4fd9a4e5aaec843ed5ae0b327f5e557c22fc6de690c39c429 WatchSource:0}: Error finding container b9be2e13878c18f4fd9a4e5aaec843ed5ae0b327f5e557c22fc6de690c39c429: Status 404 returned error can't find the container with id b9be2e13878c18f4fd9a4e5aaec843ed5ae0b327f5e557c22fc6de690c39c429 Sep 11 18:17:08.593610 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:08.593570 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-s6nf9" event={"ID":"3a406a87-d961-4352-b049-eb5476b6f7a5","Type":"ContainerStarted","Data":"b9be2e13878c18f4fd9a4e5aaec843ed5ae0b327f5e557c22fc6de690c39c429"} Sep 11 18:17:09.780829 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:09.780792 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-575bc4558d-sqmlq"] Sep 11 18:17:09.828720 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:09.828681 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-575bc4558d-sqmlq"] Sep 11 18:17:09.828914 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:09.828831 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-575bc4558d-sqmlq" Sep 11 18:17:09.831866 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:09.831837 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 11 18:17:09.832654 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:09.832632 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 11 18:17:09.832768 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:09.832675 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 11 18:17:09.832768 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:09.832675 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-nmqv8\"" Sep 11 18:17:09.832964 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:09.832946 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 11 18:17:09.833125 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:09.833002 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 11 18:17:09.833125 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:09.833011 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 11 18:17:09.833125 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:09.833030 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 11 18:17:09.968800 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:09.968764 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/95baaa6e-5c17-4be8-8b63-805a2e418838-service-ca\") pod \"console-575bc4558d-sqmlq\" (UID: \"95baaa6e-5c17-4be8-8b63-805a2e418838\") " pod="openshift-console/console-575bc4558d-sqmlq" Sep 11 18:17:09.968967 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:09.968810 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/95baaa6e-5c17-4be8-8b63-805a2e418838-console-oauth-config\") pod \"console-575bc4558d-sqmlq\" (UID: \"95baaa6e-5c17-4be8-8b63-805a2e418838\") " pod="openshift-console/console-575bc4558d-sqmlq" Sep 11 18:17:09.968967 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:09.968871 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj556\" (UniqueName: \"kubernetes.io/projected/95baaa6e-5c17-4be8-8b63-805a2e418838-kube-api-access-cj556\") pod \"console-575bc4558d-sqmlq\" (UID: \"95baaa6e-5c17-4be8-8b63-805a2e418838\") " pod="openshift-console/console-575bc4558d-sqmlq" Sep 11 18:17:09.968967 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:09.968895 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/95baaa6e-5c17-4be8-8b63-805a2e418838-oauth-serving-cert\") pod \"console-575bc4558d-sqmlq\" (UID: \"95baaa6e-5c17-4be8-8b63-805a2e418838\") " pod="openshift-console/console-575bc4558d-sqmlq" Sep 11 18:17:09.968967 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:09.968911 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/95baaa6e-5c17-4be8-8b63-805a2e418838-console-serving-cert\") pod \"console-575bc4558d-sqmlq\" (UID: \"95baaa6e-5c17-4be8-8b63-805a2e418838\") " pod="openshift-console/console-575bc4558d-sqmlq" Sep 11 18:17:09.968967 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:09.968937 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/95baaa6e-5c17-4be8-8b63-805a2e418838-console-config\") pod \"console-575bc4558d-sqmlq\" (UID: \"95baaa6e-5c17-4be8-8b63-805a2e418838\") " pod="openshift-console/console-575bc4558d-sqmlq" Sep 11 18:17:10.069777 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:10.069675 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/95baaa6e-5c17-4be8-8b63-805a2e418838-service-ca\") pod \"console-575bc4558d-sqmlq\" (UID: \"95baaa6e-5c17-4be8-8b63-805a2e418838\") " pod="openshift-console/console-575bc4558d-sqmlq" Sep 11 18:17:10.069777 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:10.069730 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/95baaa6e-5c17-4be8-8b63-805a2e418838-console-oauth-config\") pod \"console-575bc4558d-sqmlq\" (UID: \"95baaa6e-5c17-4be8-8b63-805a2e418838\") " pod="openshift-console/console-575bc4558d-sqmlq" Sep 11 18:17:10.070008 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:10.069805 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj556\" (UniqueName: \"kubernetes.io/projected/95baaa6e-5c17-4be8-8b63-805a2e418838-kube-api-access-cj556\") pod \"console-575bc4558d-sqmlq\" (UID: \"95baaa6e-5c17-4be8-8b63-805a2e418838\") " pod="openshift-console/console-575bc4558d-sqmlq" Sep 11 18:17:10.070008 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:10.069843 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/95baaa6e-5c17-4be8-8b63-805a2e418838-oauth-serving-cert\") pod \"console-575bc4558d-sqmlq\" (UID: \"95baaa6e-5c17-4be8-8b63-805a2e418838\") " pod="openshift-console/console-575bc4558d-sqmlq" Sep 11 18:17:10.070008 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:10.069867 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/95baaa6e-5c17-4be8-8b63-805a2e418838-console-serving-cert\") pod \"console-575bc4558d-sqmlq\" (UID: \"95baaa6e-5c17-4be8-8b63-805a2e418838\") " pod="openshift-console/console-575bc4558d-sqmlq" Sep 11 18:17:10.070008 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:10.069902 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/95baaa6e-5c17-4be8-8b63-805a2e418838-console-config\") pod \"console-575bc4558d-sqmlq\" (UID: \"95baaa6e-5c17-4be8-8b63-805a2e418838\") " pod="openshift-console/console-575bc4558d-sqmlq" Sep 11 18:17:10.070770 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:10.070740 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/95baaa6e-5c17-4be8-8b63-805a2e418838-console-config\") pod \"console-575bc4558d-sqmlq\" (UID: \"95baaa6e-5c17-4be8-8b63-805a2e418838\") " pod="openshift-console/console-575bc4558d-sqmlq" Sep 11 18:17:10.076937 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:10.076906 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/95baaa6e-5c17-4be8-8b63-805a2e418838-service-ca\") pod \"console-575bc4558d-sqmlq\" (UID: \"95baaa6e-5c17-4be8-8b63-805a2e418838\") " pod="openshift-console/console-575bc4558d-sqmlq" Sep 11 18:17:10.076937 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:10.076931 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/95baaa6e-5c17-4be8-8b63-805a2e418838-console-oauth-config\") pod \"console-575bc4558d-sqmlq\" (UID: \"95baaa6e-5c17-4be8-8b63-805a2e418838\") " pod="openshift-console/console-575bc4558d-sqmlq" Sep 11 18:17:10.077914 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:10.077889 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/95baaa6e-5c17-4be8-8b63-805a2e418838-oauth-serving-cert\") pod \"console-575bc4558d-sqmlq\" (UID: \"95baaa6e-5c17-4be8-8b63-805a2e418838\") " pod="openshift-console/console-575bc4558d-sqmlq" Sep 11 18:17:10.079766 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:10.079743 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj556\" (UniqueName: \"kubernetes.io/projected/95baaa6e-5c17-4be8-8b63-805a2e418838-kube-api-access-cj556\") pod \"console-575bc4558d-sqmlq\" (UID: \"95baaa6e-5c17-4be8-8b63-805a2e418838\") " pod="openshift-console/console-575bc4558d-sqmlq" Sep 11 18:17:10.079883 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:10.079864 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/95baaa6e-5c17-4be8-8b63-805a2e418838-console-serving-cert\") pod \"console-575bc4558d-sqmlq\" (UID: \"95baaa6e-5c17-4be8-8b63-805a2e418838\") " pod="openshift-console/console-575bc4558d-sqmlq" Sep 11 18:17:10.138418 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:10.138379 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-575bc4558d-sqmlq" Sep 11 18:17:10.449289 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:10.449260 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-575bc4558d-sqmlq"] Sep 11 18:17:10.601773 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:10.601737 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-575bc4558d-sqmlq" event={"ID":"95baaa6e-5c17-4be8-8b63-805a2e418838","Type":"ContainerStarted","Data":"3c3f06feb18e93d018f380bead43cfebdc791763461355708377e0c9c29553b0"} Sep 11 18:17:10.608475 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:10.608426 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-s6nf9" event={"ID":"3a406a87-d961-4352-b049-eb5476b6f7a5","Type":"ContainerStarted","Data":"662065a7966e3cfbc3d16326115efbc2505eb9fb31e52563a3e0ee23378ec4a8"} Sep 11 18:17:10.625568 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:10.625510 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-s6nf9" podStartSLOduration=33.367644001 podStartE2EDuration="35.625494615s" totalImagesPullingTime="2.257850614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:16:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:17:10.624786205 +0000 UTC m=+112.121803959" watchObservedRunningTime="2026-09-11 18:17:10.625494615 +0000 UTC m=+112.122512368" Sep 11 18:17:12.572157 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:12.572124 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-c6nk8" Sep 11 18:17:13.622434 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:13.622344 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-575bc4558d-sqmlq" event={"ID":"95baaa6e-5c17-4be8-8b63-805a2e418838","Type":"ContainerStarted","Data":"9d6c8413b317958fd223c55fb05102043d121be4ce024b7ac0ca75506d44785b"} Sep 11 18:17:13.644428 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:13.644373 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-575bc4558d-sqmlq" podStartSLOduration=1.7796380489999999 podStartE2EDuration="4.644352901s" totalImagesPullingTime="2.864714852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:17:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:17:13.643300985 +0000 UTC m=+115.140318738" watchObservedRunningTime="2026-09-11 18:17:13.644352901 +0000 UTC m=+115.141370654" Sep 11 18:17:20.138525 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:20.138491 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-575bc4558d-sqmlq" Sep 11 18:17:20.138967 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:20.138761 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-575bc4558d-sqmlq" Sep 11 18:17:20.143187 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:20.143168 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-575bc4558d-sqmlq" Sep 11 18:17:20.651581 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:20.651546 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-575bc4558d-sqmlq" Sep 11 18:17:24.579028 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:24.578996 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6cd6c4f7f7-vh7q7" Sep 11 18:17:24.860955 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:24.860873 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-csp6n"] Sep 11 18:17:24.864015 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:24.863990 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-csp6n" Sep 11 18:17:24.866461 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:24.866270 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 11 18:17:24.866461 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:24.866301 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 11 18:17:24.866461 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:24.866345 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-jrl5h\"" Sep 11 18:17:24.866461 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:24.866378 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 11 18:17:24.866743 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:24.866658 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 11 18:17:24.875334 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:24.875308 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-crss9"] Sep 11 18:17:24.877738 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:24.877707 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-csp6n"] Sep 11 18:17:24.877852 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:24.877833 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-crss9" Sep 11 18:17:24.880330 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:24.880311 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 11 18:17:24.882020 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:24.881691 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 11 18:17:24.882364 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:24.881784 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9m256\"" Sep 11 18:17:24.882639 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:24.881835 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 11 18:17:24.895211 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:24.895185 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b3d754c1-7827-4226-9abf-c2b03605bf3c-node-exporter-wtmp\") pod \"node-exporter-crss9\" (UID: \"b3d754c1-7827-4226-9abf-c2b03605bf3c\") " pod="openshift-monitoring/node-exporter-crss9" Sep 11 18:17:24.895354 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:24.895226 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc19fa81-e0df-4136-b22c-f04d63132dac-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-csp6n\" (UID: \"cc19fa81-e0df-4136-b22c-f04d63132dac\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-csp6n" Sep 11 18:17:24.895354 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:24.895253 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz5fd\" (UniqueName: \"kubernetes.io/projected/cc19fa81-e0df-4136-b22c-f04d63132dac-kube-api-access-pz5fd\") pod \"kube-state-metrics-59fbdd77d-csp6n\" (UID: \"cc19fa81-e0df-4136-b22c-f04d63132dac\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-csp6n" Sep 11 18:17:24.895354 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:24.895306 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b3d754c1-7827-4226-9abf-c2b03605bf3c-node-exporter-tls\") pod \"node-exporter-crss9\" (UID: \"b3d754c1-7827-4226-9abf-c2b03605bf3c\") " pod="openshift-monitoring/node-exporter-crss9" Sep 11 18:17:24.895621 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:24.895354 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b3d754c1-7827-4226-9abf-c2b03605bf3c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-crss9\" (UID: \"b3d754c1-7827-4226-9abf-c2b03605bf3c\") " pod="openshift-monitoring/node-exporter-crss9" Sep 11 18:17:24.895621 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:24.895385 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cc19fa81-e0df-4136-b22c-f04d63132dac-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-csp6n\" (UID: \"cc19fa81-e0df-4136-b22c-f04d63132dac\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-csp6n" Sep 11 18:17:24.895621 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:24.895476 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b3d754c1-7827-4226-9abf-c2b03605bf3c-metrics-client-ca\") pod \"node-exporter-crss9\" (UID: \"b3d754c1-7827-4226-9abf-c2b03605bf3c\") " pod="openshift-monitoring/node-exporter-crss9" Sep 11 18:17:24.895621 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:24.895516 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cc19fa81-e0df-4136-b22c-f04d63132dac-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-csp6n\" (UID: \"cc19fa81-e0df-4136-b22c-f04d63132dac\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-csp6n" Sep 11 18:17:24.895621 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:24.895542 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b3d754c1-7827-4226-9abf-c2b03605bf3c-root\") pod \"node-exporter-crss9\" (UID: \"b3d754c1-7827-4226-9abf-c2b03605bf3c\") " pod="openshift-monitoring/node-exporter-crss9" Sep 11 18:17:24.895621 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:24.895564 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/cc19fa81-e0df-4136-b22c-f04d63132dac-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-csp6n\" (UID: \"cc19fa81-e0df-4136-b22c-f04d63132dac\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-csp6n" Sep 11 18:17:24.895621 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:24.895601 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b3d754c1-7827-4226-9abf-c2b03605bf3c-sys\") pod \"node-exporter-crss9\" (UID: \"b3d754c1-7827-4226-9abf-c2b03605bf3c\") " pod="openshift-monitoring/node-exporter-crss9" Sep 11 18:17:24.895910 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:24.895624 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b3d754c1-7827-4226-9abf-c2b03605bf3c-node-exporter-textfile\") pod \"node-exporter-crss9\" (UID: \"b3d754c1-7827-4226-9abf-c2b03605bf3c\") " pod="openshift-monitoring/node-exporter-crss9" Sep 11 18:17:24.895910 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:24.895649 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jp2r\" (UniqueName: \"kubernetes.io/projected/b3d754c1-7827-4226-9abf-c2b03605bf3c-kube-api-access-5jp2r\") pod \"node-exporter-crss9\" (UID: \"b3d754c1-7827-4226-9abf-c2b03605bf3c\") " pod="openshift-monitoring/node-exporter-crss9" Sep 11 18:17:24.895910 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:24.895685 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b3d754c1-7827-4226-9abf-c2b03605bf3c-node-exporter-accelerators-collector-config\") pod \"node-exporter-crss9\" (UID: \"b3d754c1-7827-4226-9abf-c2b03605bf3c\") " pod="openshift-monitoring/node-exporter-crss9" Sep 11 18:17:24.895910 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:24.895726 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/cc19fa81-e0df-4136-b22c-f04d63132dac-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-csp6n\" (UID: \"cc19fa81-e0df-4136-b22c-f04d63132dac\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-csp6n" Sep 11 18:17:24.996150 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:24.996105 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b3d754c1-7827-4226-9abf-c2b03605bf3c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-crss9\" (UID: \"b3d754c1-7827-4226-9abf-c2b03605bf3c\") " pod="openshift-monitoring/node-exporter-crss9" Sep 11 18:17:24.996150 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:24.996154 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cc19fa81-e0df-4136-b22c-f04d63132dac-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-csp6n\" (UID: \"cc19fa81-e0df-4136-b22c-f04d63132dac\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-csp6n" Sep 11 18:17:24.996396 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:17:24.996261 2561 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Sep 11 18:17:24.996396 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:17:24.996319 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc19fa81-e0df-4136-b22c-f04d63132dac-kube-state-metrics-tls podName:cc19fa81-e0df-4136-b22c-f04d63132dac nodeName:}" failed. No retries permitted until 2026-09-11 18:17:25.496303547 +0000 UTC m=+126.993321292 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/cc19fa81-e0df-4136-b22c-f04d63132dac-kube-state-metrics-tls") pod "kube-state-metrics-59fbdd77d-csp6n" (UID: "cc19fa81-e0df-4136-b22c-f04d63132dac") : secret "kube-state-metrics-tls" not found Sep 11 18:17:24.996396 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:24.996332 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b3d754c1-7827-4226-9abf-c2b03605bf3c-metrics-client-ca\") pod \"node-exporter-crss9\" (UID: \"b3d754c1-7827-4226-9abf-c2b03605bf3c\") " pod="openshift-monitoring/node-exporter-crss9" Sep 11 18:17:24.996396 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:24.996361 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cc19fa81-e0df-4136-b22c-f04d63132dac-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-csp6n\" (UID: \"cc19fa81-e0df-4136-b22c-f04d63132dac\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-csp6n" Sep 11 18:17:24.996396 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:24.996379 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b3d754c1-7827-4226-9abf-c2b03605bf3c-root\") pod \"node-exporter-crss9\" (UID: \"b3d754c1-7827-4226-9abf-c2b03605bf3c\") " pod="openshift-monitoring/node-exporter-crss9" Sep 11 18:17:24.996661 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:24.996409 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/cc19fa81-e0df-4136-b22c-f04d63132dac-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-csp6n\" (UID: \"cc19fa81-e0df-4136-b22c-f04d63132dac\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-csp6n" Sep 11 18:17:24.996661 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:24.996455 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b3d754c1-7827-4226-9abf-c2b03605bf3c-sys\") pod \"node-exporter-crss9\" (UID: \"b3d754c1-7827-4226-9abf-c2b03605bf3c\") " pod="openshift-monitoring/node-exporter-crss9" Sep 11 18:17:24.996661 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:24.996479 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b3d754c1-7827-4226-9abf-c2b03605bf3c-node-exporter-textfile\") pod \"node-exporter-crss9\" (UID: \"b3d754c1-7827-4226-9abf-c2b03605bf3c\") " pod="openshift-monitoring/node-exporter-crss9" Sep 11 18:17:24.996661 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:24.996502 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jp2r\" (UniqueName: \"kubernetes.io/projected/b3d754c1-7827-4226-9abf-c2b03605bf3c-kube-api-access-5jp2r\") pod \"node-exporter-crss9\" (UID: \"b3d754c1-7827-4226-9abf-c2b03605bf3c\") " pod="openshift-monitoring/node-exporter-crss9" Sep 11 18:17:24.996661 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:24.996547 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b3d754c1-7827-4226-9abf-c2b03605bf3c-node-exporter-accelerators-collector-config\") pod \"node-exporter-crss9\" (UID: \"b3d754c1-7827-4226-9abf-c2b03605bf3c\") " pod="openshift-monitoring/node-exporter-crss9" Sep 11 18:17:24.996661 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:24.996545 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b3d754c1-7827-4226-9abf-c2b03605bf3c-root\") pod \"node-exporter-crss9\" (UID: \"b3d754c1-7827-4226-9abf-c2b03605bf3c\") " pod="openshift-monitoring/node-exporter-crss9" Sep 11 18:17:24.996661 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:24.996594 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b3d754c1-7827-4226-9abf-c2b03605bf3c-sys\") pod \"node-exporter-crss9\" (UID: \"b3d754c1-7827-4226-9abf-c2b03605bf3c\") " pod="openshift-monitoring/node-exporter-crss9" Sep 11 18:17:24.997021 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:24.996823 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b3d754c1-7827-4226-9abf-c2b03605bf3c-node-exporter-textfile\") pod \"node-exporter-crss9\" (UID: \"b3d754c1-7827-4226-9abf-c2b03605bf3c\") " pod="openshift-monitoring/node-exporter-crss9" Sep 11 18:17:24.997021 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:24.996891 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/cc19fa81-e0df-4136-b22c-f04d63132dac-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-csp6n\" (UID: \"cc19fa81-e0df-4136-b22c-f04d63132dac\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-csp6n" Sep 11 18:17:24.997021 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:24.996942 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b3d754c1-7827-4226-9abf-c2b03605bf3c-node-exporter-wtmp\") pod \"node-exporter-crss9\" (UID: \"b3d754c1-7827-4226-9abf-c2b03605bf3c\") " pod="openshift-monitoring/node-exporter-crss9" Sep 11 18:17:24.997021 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:24.997003 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc19fa81-e0df-4136-b22c-f04d63132dac-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-csp6n\" (UID: \"cc19fa81-e0df-4136-b22c-f04d63132dac\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-csp6n" Sep 11 18:17:24.997021 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:24.997007 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b3d754c1-7827-4226-9abf-c2b03605bf3c-metrics-client-ca\") pod \"node-exporter-crss9\" (UID: \"b3d754c1-7827-4226-9abf-c2b03605bf3c\") " pod="openshift-monitoring/node-exporter-crss9" Sep 11 18:17:24.997303 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:24.997041 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz5fd\" (UniqueName: \"kubernetes.io/projected/cc19fa81-e0df-4136-b22c-f04d63132dac-kube-api-access-pz5fd\") pod \"kube-state-metrics-59fbdd77d-csp6n\" (UID: \"cc19fa81-e0df-4136-b22c-f04d63132dac\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-csp6n" Sep 11 18:17:24.997303 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:24.997100 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b3d754c1-7827-4226-9abf-c2b03605bf3c-node-exporter-tls\") pod \"node-exporter-crss9\" (UID: \"b3d754c1-7827-4226-9abf-c2b03605bf3c\") " pod="openshift-monitoring/node-exporter-crss9" Sep 11 18:17:24.997303 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:24.997190 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b3d754c1-7827-4226-9abf-c2b03605bf3c-node-exporter-wtmp\") pod \"node-exporter-crss9\" (UID: \"b3d754c1-7827-4226-9abf-c2b03605bf3c\") " pod="openshift-monitoring/node-exporter-crss9" Sep 11 18:17:24.997467 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:24.997322 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b3d754c1-7827-4226-9abf-c2b03605bf3c-node-exporter-accelerators-collector-config\") pod \"node-exporter-crss9\" (UID: \"b3d754c1-7827-4226-9abf-c2b03605bf3c\") " pod="openshift-monitoring/node-exporter-crss9" Sep 11 18:17:24.997546 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:24.997525 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/cc19fa81-e0df-4136-b22c-f04d63132dac-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-csp6n\" (UID: \"cc19fa81-e0df-4136-b22c-f04d63132dac\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-csp6n" Sep 11 18:17:24.997799 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:24.997769 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/cc19fa81-e0df-4136-b22c-f04d63132dac-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-csp6n\" (UID: \"cc19fa81-e0df-4136-b22c-f04d63132dac\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-csp6n" Sep 11 18:17:24.997922 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:24.997894 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc19fa81-e0df-4136-b22c-f04d63132dac-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-csp6n\" (UID: \"cc19fa81-e0df-4136-b22c-f04d63132dac\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-csp6n" Sep 11 18:17:24.999009 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:24.998987 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b3d754c1-7827-4226-9abf-c2b03605bf3c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-crss9\" (UID: \"b3d754c1-7827-4226-9abf-c2b03605bf3c\") " pod="openshift-monitoring/node-exporter-crss9" Sep 11 18:17:24.999141 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:24.999118 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cc19fa81-e0df-4136-b22c-f04d63132dac-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-csp6n\" (UID: \"cc19fa81-e0df-4136-b22c-f04d63132dac\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-csp6n" Sep 11 18:17:24.999643 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:24.999623 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b3d754c1-7827-4226-9abf-c2b03605bf3c-node-exporter-tls\") pod \"node-exporter-crss9\" (UID: \"b3d754c1-7827-4226-9abf-c2b03605bf3c\") " pod="openshift-monitoring/node-exporter-crss9" Sep 11 18:17:25.009918 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:25.009890 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jp2r\" (UniqueName: \"kubernetes.io/projected/b3d754c1-7827-4226-9abf-c2b03605bf3c-kube-api-access-5jp2r\") pod \"node-exporter-crss9\" (UID: \"b3d754c1-7827-4226-9abf-c2b03605bf3c\") " pod="openshift-monitoring/node-exporter-crss9" Sep 11 18:17:25.013013 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:25.012985 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz5fd\" (UniqueName: \"kubernetes.io/projected/cc19fa81-e0df-4136-b22c-f04d63132dac-kube-api-access-pz5fd\") pod \"kube-state-metrics-59fbdd77d-csp6n\" (UID: \"cc19fa81-e0df-4136-b22c-f04d63132dac\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-csp6n" Sep 11 18:17:25.189557 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:25.189515 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-crss9" Sep 11 18:17:25.505019 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:25.504929 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cc19fa81-e0df-4136-b22c-f04d63132dac-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-csp6n\" (UID: \"cc19fa81-e0df-4136-b22c-f04d63132dac\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-csp6n" Sep 11 18:17:25.507357 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:25.507334 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cc19fa81-e0df-4136-b22c-f04d63132dac-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-csp6n\" (UID: \"cc19fa81-e0df-4136-b22c-f04d63132dac\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-csp6n" Sep 11 18:17:25.666356 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:25.666319 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-crss9" event={"ID":"b3d754c1-7827-4226-9abf-c2b03605bf3c","Type":"ContainerStarted","Data":"7f9407c52387a8effe00932eed10d46c867b2b19688d4fbc582faae3b2097091"} Sep 11 18:17:25.773960 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:25.773871 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-csp6n" Sep 11 18:17:26.064939 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:26.064911 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-csp6n"] Sep 11 18:17:26.185263 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:17:26.185225 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc19fa81_e0df_4136_b22c_f04d63132dac.slice/crio-4f72b9746bc07e092665900acdc06b254c89958ab5b0210bb6cc96eb738ebbb0 WatchSource:0}: Error finding container 4f72b9746bc07e092665900acdc06b254c89958ab5b0210bb6cc96eb738ebbb0: Status 404 returned error can't find the container with id 4f72b9746bc07e092665900acdc06b254c89958ab5b0210bb6cc96eb738ebbb0 Sep 11 18:17:26.671451 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:26.671416 2561 generic.go:356] "Generic (PLEG): container finished" podID="b3d754c1-7827-4226-9abf-c2b03605bf3c" containerID="69191880ef9bf29da01f196ce7f6caaebc665f471e345375a43515413173093c" exitCode=0 Sep 11 18:17:26.671910 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:26.671508 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-crss9" event={"ID":"b3d754c1-7827-4226-9abf-c2b03605bf3c","Type":"ContainerDied","Data":"69191880ef9bf29da01f196ce7f6caaebc665f471e345375a43515413173093c"} Sep 11 18:17:26.672838 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:26.672810 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-csp6n" event={"ID":"cc19fa81-e0df-4136-b22c-f04d63132dac","Type":"ContainerStarted","Data":"4f72b9746bc07e092665900acdc06b254c89958ab5b0210bb6cc96eb738ebbb0"} Sep 11 18:17:27.429672 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:27.429628 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-575bc4558d-sqmlq"] Sep 11 18:17:27.681021 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:27.680868 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-crss9" event={"ID":"b3d754c1-7827-4226-9abf-c2b03605bf3c","Type":"ContainerStarted","Data":"caea57ab9d479ba1b88e4e6b1ebed2d7466596d226c7daba795dc70cc0cb34fe"} Sep 11 18:17:27.681021 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:27.680956 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-crss9" event={"ID":"b3d754c1-7827-4226-9abf-c2b03605bf3c","Type":"ContainerStarted","Data":"0331dbd9d789e9b03be33fe3b14a615a1cbda4c6051abb786b7d12414ea7b96e"} Sep 11 18:17:27.684777 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:27.684740 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-csp6n" event={"ID":"cc19fa81-e0df-4136-b22c-f04d63132dac","Type":"ContainerStarted","Data":"bfaaa3b48dabe06cd8052ab91b98bd5bea1cc678d85708fa16f08fad5ea40ea3"} Sep 11 18:17:27.684911 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:27.684819 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-csp6n" event={"ID":"cc19fa81-e0df-4136-b22c-f04d63132dac","Type":"ContainerStarted","Data":"15df4c80444e80271914be919ad3f0cabae19df788c4efe244b7d282bebd339a"} Sep 11 18:17:27.707665 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:27.707613 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-crss9" podStartSLOduration=2.639017783 podStartE2EDuration="3.707593129s" totalImagesPullingTime="1.000442213s" totalInitContainerRuntime="68.133133ms" isStatefulPod=true podCreationTimestamp="2026-09-11 18:17:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:17:27.705499751 +0000 UTC m=+129.202517506" watchObservedRunningTime="2026-09-11 18:17:27.707593129 +0000 UTC m=+129.204610877" Sep 11 18:17:28.691677 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:28.691642 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-csp6n" event={"ID":"cc19fa81-e0df-4136-b22c-f04d63132dac","Type":"ContainerStarted","Data":"44c6bf6b9b13b935b765579de522e3ba43ee3b999aad4497edd738eb50ce5bfd"} Sep 11 18:17:28.735285 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:28.735002 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d-metrics-certs\") pod \"network-metrics-daemon-cmf6c\" (UID: \"a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d\") " pod="openshift-multus/network-metrics-daemon-cmf6c" Sep 11 18:17:28.737365 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:28.737337 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d-metrics-certs\") pod \"network-metrics-daemon-cmf6c\" (UID: \"a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d\") " pod="openshift-multus/network-metrics-daemon-cmf6c" Sep 11 18:17:28.948477 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:28.948395 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-28rql\"" Sep 11 18:17:28.957165 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:28.957136 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cmf6c" Sep 11 18:17:29.230986 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:29.230922 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-csp6n" podStartSLOduration=3.914562701 podStartE2EDuration="5.230897174s" totalImagesPullingTime="1.316334473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:17:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:17:28.71060853 +0000 UTC m=+130.207626286" watchObservedRunningTime="2026-09-11 18:17:29.230897174 +0000 UTC m=+130.727914929" Sep 11 18:17:29.231716 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:29.231489 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cmf6c"] Sep 11 18:17:29.696910 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:29.696873 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cmf6c" event={"ID":"a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d","Type":"ContainerStarted","Data":"60fe0a943229baf18799ef650e728b974e6132e0ae6d355634491dbc53d177a1"} Sep 11 18:17:30.702668 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:30.702627 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cmf6c" event={"ID":"a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d","Type":"ContainerStarted","Data":"e2ba51aa34f30b11ce6ecfa824bdedcc34eece24fbdddde2a79f64a699f256db"} Sep 11 18:17:30.703085 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:30.702693 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cmf6c" event={"ID":"a1a215bc-d1ee-4b11-b29f-70b9ca9fce3d","Type":"ContainerStarted","Data":"f8bc0cecc3a92c7ceec197b07db205b5902825f9e702258764b46114f856eef0"} Sep 11 18:17:30.726439 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:30.726379 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-cmf6c" podStartSLOduration=130.755669674 podStartE2EDuration="2m11.726360503s" totalImagesPullingTime="970.690829ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:15:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:17:30.724391295 +0000 UTC m=+132.221409048" watchObservedRunningTime="2026-09-11 18:17:30.726360503 +0000 UTC m=+132.223378257" Sep 11 18:17:33.932366 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:33.932329 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-586659979-qlk7m"] Sep 11 18:17:33.935151 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:33.935123 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-586659979-qlk7m" Sep 11 18:17:33.942395 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:33.942371 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 11 18:17:33.945453 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:33.945428 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-586659979-qlk7m"] Sep 11 18:17:33.978725 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:33.978683 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6f85e5c5-b612-4d6e-ac5a-4dca63a8980d-service-ca\") pod \"console-586659979-qlk7m\" (UID: \"6f85e5c5-b612-4d6e-ac5a-4dca63a8980d\") " pod="openshift-console/console-586659979-qlk7m" Sep 11 18:17:33.978912 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:33.978747 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6f85e5c5-b612-4d6e-ac5a-4dca63a8980d-oauth-serving-cert\") pod \"console-586659979-qlk7m\" (UID: \"6f85e5c5-b612-4d6e-ac5a-4dca63a8980d\") " pod="openshift-console/console-586659979-qlk7m" Sep 11 18:17:33.978912 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:33.978811 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f85e5c5-b612-4d6e-ac5a-4dca63a8980d-trusted-ca-bundle\") pod \"console-586659979-qlk7m\" (UID: \"6f85e5c5-b612-4d6e-ac5a-4dca63a8980d\") " pod="openshift-console/console-586659979-qlk7m" Sep 11 18:17:33.978912 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:33.978837 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6f85e5c5-b612-4d6e-ac5a-4dca63a8980d-console-oauth-config\") pod \"console-586659979-qlk7m\" (UID: \"6f85e5c5-b612-4d6e-ac5a-4dca63a8980d\") " pod="openshift-console/console-586659979-qlk7m" Sep 11 18:17:33.978912 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:33.978892 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6f85e5c5-b612-4d6e-ac5a-4dca63a8980d-console-config\") pod \"console-586659979-qlk7m\" (UID: \"6f85e5c5-b612-4d6e-ac5a-4dca63a8980d\") " pod="openshift-console/console-586659979-qlk7m" Sep 11 18:17:33.978912 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:33.978914 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q98qx\" (UniqueName: \"kubernetes.io/projected/6f85e5c5-b612-4d6e-ac5a-4dca63a8980d-kube-api-access-q98qx\") pod \"console-586659979-qlk7m\" (UID: \"6f85e5c5-b612-4d6e-ac5a-4dca63a8980d\") " pod="openshift-console/console-586659979-qlk7m" Sep 11 18:17:33.979146 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:33.978940 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6f85e5c5-b612-4d6e-ac5a-4dca63a8980d-console-serving-cert\") pod \"console-586659979-qlk7m\" (UID: \"6f85e5c5-b612-4d6e-ac5a-4dca63a8980d\") " pod="openshift-console/console-586659979-qlk7m" Sep 11 18:17:34.080027 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:34.079985 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6f85e5c5-b612-4d6e-ac5a-4dca63a8980d-oauth-serving-cert\") pod \"console-586659979-qlk7m\" (UID: \"6f85e5c5-b612-4d6e-ac5a-4dca63a8980d\") " pod="openshift-console/console-586659979-qlk7m" Sep 11 18:17:34.080252 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:34.080047 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f85e5c5-b612-4d6e-ac5a-4dca63a8980d-trusted-ca-bundle\") pod \"console-586659979-qlk7m\" (UID: \"6f85e5c5-b612-4d6e-ac5a-4dca63a8980d\") " pod="openshift-console/console-586659979-qlk7m" Sep 11 18:17:34.080252 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:34.080103 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6f85e5c5-b612-4d6e-ac5a-4dca63a8980d-console-oauth-config\") pod \"console-586659979-qlk7m\" (UID: \"6f85e5c5-b612-4d6e-ac5a-4dca63a8980d\") " pod="openshift-console/console-586659979-qlk7m" Sep 11 18:17:34.080252 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:34.080145 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6f85e5c5-b612-4d6e-ac5a-4dca63a8980d-console-config\") pod \"console-586659979-qlk7m\" (UID: \"6f85e5c5-b612-4d6e-ac5a-4dca63a8980d\") " pod="openshift-console/console-586659979-qlk7m" Sep 11 18:17:34.080252 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:34.080171 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q98qx\" (UniqueName: \"kubernetes.io/projected/6f85e5c5-b612-4d6e-ac5a-4dca63a8980d-kube-api-access-q98qx\") pod \"console-586659979-qlk7m\" (UID: \"6f85e5c5-b612-4d6e-ac5a-4dca63a8980d\") " pod="openshift-console/console-586659979-qlk7m" Sep 11 18:17:34.080252 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:34.080196 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6f85e5c5-b612-4d6e-ac5a-4dca63a8980d-console-serving-cert\") pod \"console-586659979-qlk7m\" (UID: \"6f85e5c5-b612-4d6e-ac5a-4dca63a8980d\") " pod="openshift-console/console-586659979-qlk7m" Sep 11 18:17:34.080252 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:34.080242 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6f85e5c5-b612-4d6e-ac5a-4dca63a8980d-service-ca\") pod \"console-586659979-qlk7m\" (UID: \"6f85e5c5-b612-4d6e-ac5a-4dca63a8980d\") " pod="openshift-console/console-586659979-qlk7m" Sep 11 18:17:34.080929 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:34.080902 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6f85e5c5-b612-4d6e-ac5a-4dca63a8980d-console-config\") pod \"console-586659979-qlk7m\" (UID: \"6f85e5c5-b612-4d6e-ac5a-4dca63a8980d\") " pod="openshift-console/console-586659979-qlk7m" Sep 11 18:17:34.081027 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:34.080907 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6f85e5c5-b612-4d6e-ac5a-4dca63a8980d-service-ca\") pod \"console-586659979-qlk7m\" (UID: \"6f85e5c5-b612-4d6e-ac5a-4dca63a8980d\") " pod="openshift-console/console-586659979-qlk7m" Sep 11 18:17:34.081027 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:34.080907 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6f85e5c5-b612-4d6e-ac5a-4dca63a8980d-oauth-serving-cert\") pod \"console-586659979-qlk7m\" (UID: \"6f85e5c5-b612-4d6e-ac5a-4dca63a8980d\") " pod="openshift-console/console-586659979-qlk7m" Sep 11 18:17:34.081027 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:34.080992 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f85e5c5-b612-4d6e-ac5a-4dca63a8980d-trusted-ca-bundle\") pod \"console-586659979-qlk7m\" (UID: \"6f85e5c5-b612-4d6e-ac5a-4dca63a8980d\") " pod="openshift-console/console-586659979-qlk7m" Sep 11 18:17:34.083192 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:34.083170 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6f85e5c5-b612-4d6e-ac5a-4dca63a8980d-console-serving-cert\") pod \"console-586659979-qlk7m\" (UID: \"6f85e5c5-b612-4d6e-ac5a-4dca63a8980d\") " pod="openshift-console/console-586659979-qlk7m" Sep 11 18:17:34.083192 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:34.083182 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6f85e5c5-b612-4d6e-ac5a-4dca63a8980d-console-oauth-config\") pod \"console-586659979-qlk7m\" (UID: \"6f85e5c5-b612-4d6e-ac5a-4dca63a8980d\") " pod="openshift-console/console-586659979-qlk7m" Sep 11 18:17:34.089307 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:34.089282 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q98qx\" (UniqueName: \"kubernetes.io/projected/6f85e5c5-b612-4d6e-ac5a-4dca63a8980d-kube-api-access-q98qx\") pod \"console-586659979-qlk7m\" (UID: \"6f85e5c5-b612-4d6e-ac5a-4dca63a8980d\") " pod="openshift-console/console-586659979-qlk7m" Sep 11 18:17:34.245437 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:34.245350 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-586659979-qlk7m" Sep 11 18:17:34.515892 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:34.515809 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-586659979-qlk7m"] Sep 11 18:17:34.518685 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:17:34.518655 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f85e5c5_b612_4d6e_ac5a_4dca63a8980d.slice/crio-f65e19056f87a007f0abdcc4d209891ae53d87094198b3c6bdc377210832a0b4 WatchSource:0}: Error finding container f65e19056f87a007f0abdcc4d209891ae53d87094198b3c6bdc377210832a0b4: Status 404 returned error can't find the container with id f65e19056f87a007f0abdcc4d209891ae53d87094198b3c6bdc377210832a0b4 Sep 11 18:17:34.719166 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:34.719126 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-586659979-qlk7m" event={"ID":"6f85e5c5-b612-4d6e-ac5a-4dca63a8980d","Type":"ContainerStarted","Data":"2c4cfb6f91e58433bf0a400095ba1ba51da509266e8771c5f3e29087753cc3b9"} Sep 11 18:17:34.719356 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:34.719186 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-586659979-qlk7m" event={"ID":"6f85e5c5-b612-4d6e-ac5a-4dca63a8980d","Type":"ContainerStarted","Data":"f65e19056f87a007f0abdcc4d209891ae53d87094198b3c6bdc377210832a0b4"} Sep 11 18:17:34.737305 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:34.737254 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-586659979-qlk7m" podStartSLOduration=1.7372357840000001 podStartE2EDuration="1.737235784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:17:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:17:34.735668153 +0000 UTC m=+136.232685919" watchObservedRunningTime="2026-09-11 18:17:34.737235784 +0000 UTC m=+136.234253538" Sep 11 18:17:44.245829 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:44.245793 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-586659979-qlk7m" Sep 11 18:17:44.246338 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:44.245870 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-586659979-qlk7m" Sep 11 18:17:44.250400 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:44.250371 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-586659979-qlk7m" Sep 11 18:17:44.758737 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:44.758706 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-586659979-qlk7m" Sep 11 18:17:45.760020 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:45.759985 2561 generic.go:356] "Generic (PLEG): container finished" podID="73a83e98-5fd7-43a6-85cc-ffd8d36057cf" containerID="a482987bc35b6d92c0bdfddc2373c13edb8367c8d439b87345a317d74394d7b6" exitCode=0 Sep 11 18:17:45.760425 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:45.760070 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9pjz5" event={"ID":"73a83e98-5fd7-43a6-85cc-ffd8d36057cf","Type":"ContainerDied","Data":"a482987bc35b6d92c0bdfddc2373c13edb8367c8d439b87345a317d74394d7b6"} Sep 11 18:17:45.760513 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:45.760498 2561 scope.go:117] "RemoveContainer" containerID="a482987bc35b6d92c0bdfddc2373c13edb8367c8d439b87345a317d74394d7b6" Sep 11 18:17:46.765679 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:46.765644 2561 generic.go:356] "Generic (PLEG): container finished" podID="6bdab508-3741-4fe3-8f60-233f57470db8" containerID="4dbec84b86d46c7e582fa9ffbbc7b1b38c2fe18ec118359c35741c5d673f1050" exitCode=0 Sep 11 18:17:46.766108 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:46.765720 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2szfj" event={"ID":"6bdab508-3741-4fe3-8f60-233f57470db8","Type":"ContainerDied","Data":"4dbec84b86d46c7e582fa9ffbbc7b1b38c2fe18ec118359c35741c5d673f1050"} Sep 11 18:17:46.766202 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:46.766176 2561 scope.go:117] "RemoveContainer" containerID="4dbec84b86d46c7e582fa9ffbbc7b1b38c2fe18ec118359c35741c5d673f1050" Sep 11 18:17:46.767695 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:46.767673 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9pjz5" event={"ID":"73a83e98-5fd7-43a6-85cc-ffd8d36057cf","Type":"ContainerStarted","Data":"2716f4a4dce7d7d156dd031e2a3c58ade1572c247f122abd0253b95c0217e3e6"} Sep 11 18:17:47.773767 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:47.773735 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2szfj" event={"ID":"6bdab508-3741-4fe3-8f60-233f57470db8","Type":"ContainerStarted","Data":"db29f3f44c5d6a2e3b1c027b2ec731a734e718b0ffad69df007d84af17ea783f"} Sep 11 18:17:52.451478 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:52.451405 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-575bc4558d-sqmlq" podUID="95baaa6e-5c17-4be8-8b63-805a2e418838" containerName="console" containerID="cri-o://9d6c8413b317958fd223c55fb05102043d121be4ce024b7ac0ca75506d44785b" gracePeriod=15 Sep 11 18:17:52.693490 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:52.693466 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-575bc4558d-sqmlq" Sep 11 18:17:52.798149 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:52.798012 2561 generic.go:356] "Generic (PLEG): container finished" podID="95baaa6e-5c17-4be8-8b63-805a2e418838" containerID="9d6c8413b317958fd223c55fb05102043d121be4ce024b7ac0ca75506d44785b" exitCode=0 Sep 11 18:17:52.798149 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:52.798129 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-575bc4558d-sqmlq" Sep 11 18:17:52.798336 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:52.798133 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-575bc4558d-sqmlq" event={"ID":"95baaa6e-5c17-4be8-8b63-805a2e418838","Type":"ContainerDied","Data":"9d6c8413b317958fd223c55fb05102043d121be4ce024b7ac0ca75506d44785b"} Sep 11 18:17:52.798336 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:52.798313 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-575bc4558d-sqmlq" event={"ID":"95baaa6e-5c17-4be8-8b63-805a2e418838","Type":"ContainerDied","Data":"3c3f06feb18e93d018f380bead43cfebdc791763461355708377e0c9c29553b0"} Sep 11 18:17:52.798413 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:52.798367 2561 scope.go:117] "RemoveContainer" containerID="9d6c8413b317958fd223c55fb05102043d121be4ce024b7ac0ca75506d44785b" Sep 11 18:17:52.808264 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:52.808240 2561 scope.go:117] "RemoveContainer" containerID="9d6c8413b317958fd223c55fb05102043d121be4ce024b7ac0ca75506d44785b" Sep 11 18:17:52.808597 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:17:52.808573 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d6c8413b317958fd223c55fb05102043d121be4ce024b7ac0ca75506d44785b\": container with ID starting with 9d6c8413b317958fd223c55fb05102043d121be4ce024b7ac0ca75506d44785b not found: ID does not exist" containerID="9d6c8413b317958fd223c55fb05102043d121be4ce024b7ac0ca75506d44785b" Sep 11 18:17:52.808658 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:52.808605 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d6c8413b317958fd223c55fb05102043d121be4ce024b7ac0ca75506d44785b"} err="failed to get container status \"9d6c8413b317958fd223c55fb05102043d121be4ce024b7ac0ca75506d44785b\": rpc error: code = NotFound desc = could not find container \"9d6c8413b317958fd223c55fb05102043d121be4ce024b7ac0ca75506d44785b\": container with ID starting with 9d6c8413b317958fd223c55fb05102043d121be4ce024b7ac0ca75506d44785b not found: ID does not exist" Sep 11 18:17:52.851132 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:52.851092 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cj556\" (UniqueName: \"kubernetes.io/projected/95baaa6e-5c17-4be8-8b63-805a2e418838-kube-api-access-cj556\") pod \"95baaa6e-5c17-4be8-8b63-805a2e418838\" (UID: \"95baaa6e-5c17-4be8-8b63-805a2e418838\") " Sep 11 18:17:52.851269 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:52.851166 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/95baaa6e-5c17-4be8-8b63-805a2e418838-console-oauth-config\") pod \"95baaa6e-5c17-4be8-8b63-805a2e418838\" (UID: \"95baaa6e-5c17-4be8-8b63-805a2e418838\") " Sep 11 18:17:52.851269 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:52.851189 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/95baaa6e-5c17-4be8-8b63-805a2e418838-console-serving-cert\") pod \"95baaa6e-5c17-4be8-8b63-805a2e418838\" (UID: \"95baaa6e-5c17-4be8-8b63-805a2e418838\") " Sep 11 18:17:52.851269 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:52.851229 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/95baaa6e-5c17-4be8-8b63-805a2e418838-console-config\") pod \"95baaa6e-5c17-4be8-8b63-805a2e418838\" (UID: \"95baaa6e-5c17-4be8-8b63-805a2e418838\") " Sep 11 18:17:52.851269 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:52.851261 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/95baaa6e-5c17-4be8-8b63-805a2e418838-service-ca\") pod \"95baaa6e-5c17-4be8-8b63-805a2e418838\" (UID: \"95baaa6e-5c17-4be8-8b63-805a2e418838\") " Sep 11 18:17:52.851440 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:52.851283 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/95baaa6e-5c17-4be8-8b63-805a2e418838-oauth-serving-cert\") pod \"95baaa6e-5c17-4be8-8b63-805a2e418838\" (UID: \"95baaa6e-5c17-4be8-8b63-805a2e418838\") " Sep 11 18:17:52.851737 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:52.851703 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95baaa6e-5c17-4be8-8b63-805a2e418838-console-config" (OuterVolumeSpecName: "console-config") pod "95baaa6e-5c17-4be8-8b63-805a2e418838" (UID: "95baaa6e-5c17-4be8-8b63-805a2e418838"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:17:52.851737 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:52.851722 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95baaa6e-5c17-4be8-8b63-805a2e418838-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "95baaa6e-5c17-4be8-8b63-805a2e418838" (UID: "95baaa6e-5c17-4be8-8b63-805a2e418838"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:17:52.851913 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:52.851709 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95baaa6e-5c17-4be8-8b63-805a2e418838-service-ca" (OuterVolumeSpecName: "service-ca") pod "95baaa6e-5c17-4be8-8b63-805a2e418838" (UID: "95baaa6e-5c17-4be8-8b63-805a2e418838"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:17:52.853584 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:52.853547 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95baaa6e-5c17-4be8-8b63-805a2e418838-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "95baaa6e-5c17-4be8-8b63-805a2e418838" (UID: "95baaa6e-5c17-4be8-8b63-805a2e418838"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:17:52.853690 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:52.853586 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95baaa6e-5c17-4be8-8b63-805a2e418838-kube-api-access-cj556" (OuterVolumeSpecName: "kube-api-access-cj556") pod "95baaa6e-5c17-4be8-8b63-805a2e418838" (UID: "95baaa6e-5c17-4be8-8b63-805a2e418838"). InnerVolumeSpecName "kube-api-access-cj556". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:17:52.853690 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:52.853595 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95baaa6e-5c17-4be8-8b63-805a2e418838-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "95baaa6e-5c17-4be8-8b63-805a2e418838" (UID: "95baaa6e-5c17-4be8-8b63-805a2e418838"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:17:52.952395 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:52.952356 2561 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/95baaa6e-5c17-4be8-8b63-805a2e418838-console-oauth-config\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:17:52.952395 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:52.952388 2561 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/95baaa6e-5c17-4be8-8b63-805a2e418838-console-serving-cert\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:17:52.952395 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:52.952399 2561 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/95baaa6e-5c17-4be8-8b63-805a2e418838-console-config\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:17:52.952395 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:52.952408 2561 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/95baaa6e-5c17-4be8-8b63-805a2e418838-service-ca\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:17:52.952703 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:52.952417 2561 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/95baaa6e-5c17-4be8-8b63-805a2e418838-oauth-serving-cert\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:17:52.952703 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:52.952426 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj556\" (UniqueName: \"kubernetes.io/projected/95baaa6e-5c17-4be8-8b63-805a2e418838-kube-api-access-cj556\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:17:53.120548 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:53.120507 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-575bc4558d-sqmlq"] Sep 11 18:17:53.124955 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:53.124925 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-575bc4558d-sqmlq"] Sep 11 18:17:53.137566 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:17:53.137539 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95baaa6e-5c17-4be8-8b63-805a2e418838" path="/var/lib/kubelet/pods/95baaa6e-5c17-4be8-8b63-805a2e418838/volumes" Sep 11 18:18:00.830762 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:18:00.830725 2561 generic.go:356] "Generic (PLEG): container finished" podID="97808c4d-b62a-4ca1-b686-5e27a9aa79a0" containerID="08bcb5c73db4fa45a4933cd84c72c127b7df9ed99a1fd914f3dee49f652ce9ec" exitCode=0 Sep 11 18:18:00.831193 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:18:00.830777 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-spcnr" event={"ID":"97808c4d-b62a-4ca1-b686-5e27a9aa79a0","Type":"ContainerDied","Data":"08bcb5c73db4fa45a4933cd84c72c127b7df9ed99a1fd914f3dee49f652ce9ec"} Sep 11 18:18:00.831193 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:18:00.831128 2561 scope.go:117] "RemoveContainer" containerID="08bcb5c73db4fa45a4933cd84c72c127b7df9ed99a1fd914f3dee49f652ce9ec" Sep 11 18:18:01.836034 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:18:01.835999 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-spcnr" event={"ID":"97808c4d-b62a-4ca1-b686-5e27a9aa79a0","Type":"ContainerStarted","Data":"6a429580a6f5f61a729a283b5c8ef4b34dd026fbcec32021fb065a56a2e99615"} Sep 11 18:18:51.896395 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:18:51.896358 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-978dfdd6c-4s4qq"] Sep 11 18:18:51.896885 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:18:51.896675 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95baaa6e-5c17-4be8-8b63-805a2e418838" containerName="console" Sep 11 18:18:51.896885 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:18:51.896691 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="95baaa6e-5c17-4be8-8b63-805a2e418838" containerName="console" Sep 11 18:18:51.896885 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:18:51.896747 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="95baaa6e-5c17-4be8-8b63-805a2e418838" containerName="console" Sep 11 18:18:51.899466 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:18:51.899445 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-978dfdd6c-4s4qq" Sep 11 18:18:51.910638 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:18:51.910613 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-978dfdd6c-4s4qq"] Sep 11 18:18:52.026987 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:18:52.026948 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d10f5f4-8804-4de1-9245-9ada4d29efd7-service-ca\") pod \"console-978dfdd6c-4s4qq\" (UID: \"1d10f5f4-8804-4de1-9245-9ada4d29efd7\") " pod="openshift-console/console-978dfdd6c-4s4qq" Sep 11 18:18:52.027182 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:18:52.027000 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d10f5f4-8804-4de1-9245-9ada4d29efd7-oauth-serving-cert\") pod \"console-978dfdd6c-4s4qq\" (UID: \"1d10f5f4-8804-4de1-9245-9ada4d29efd7\") " pod="openshift-console/console-978dfdd6c-4s4qq" Sep 11 18:18:52.027182 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:18:52.027019 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d10f5f4-8804-4de1-9245-9ada4d29efd7-console-config\") pod \"console-978dfdd6c-4s4qq\" (UID: \"1d10f5f4-8804-4de1-9245-9ada4d29efd7\") " pod="openshift-console/console-978dfdd6c-4s4qq" Sep 11 18:18:52.027182 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:18:52.027049 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d10f5f4-8804-4de1-9245-9ada4d29efd7-console-serving-cert\") pod \"console-978dfdd6c-4s4qq\" (UID: \"1d10f5f4-8804-4de1-9245-9ada4d29efd7\") " pod="openshift-console/console-978dfdd6c-4s4qq" Sep 11 18:18:52.027182 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:18:52.027086 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d10f5f4-8804-4de1-9245-9ada4d29efd7-trusted-ca-bundle\") pod \"console-978dfdd6c-4s4qq\" (UID: \"1d10f5f4-8804-4de1-9245-9ada4d29efd7\") " pod="openshift-console/console-978dfdd6c-4s4qq" Sep 11 18:18:52.027182 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:18:52.027132 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skqnr\" (UniqueName: \"kubernetes.io/projected/1d10f5f4-8804-4de1-9245-9ada4d29efd7-kube-api-access-skqnr\") pod \"console-978dfdd6c-4s4qq\" (UID: \"1d10f5f4-8804-4de1-9245-9ada4d29efd7\") " pod="openshift-console/console-978dfdd6c-4s4qq" Sep 11 18:18:52.027182 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:18:52.027177 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d10f5f4-8804-4de1-9245-9ada4d29efd7-console-oauth-config\") pod \"console-978dfdd6c-4s4qq\" (UID: \"1d10f5f4-8804-4de1-9245-9ada4d29efd7\") " pod="openshift-console/console-978dfdd6c-4s4qq" Sep 11 18:18:52.128341 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:18:52.128303 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skqnr\" (UniqueName: \"kubernetes.io/projected/1d10f5f4-8804-4de1-9245-9ada4d29efd7-kube-api-access-skqnr\") pod \"console-978dfdd6c-4s4qq\" (UID: \"1d10f5f4-8804-4de1-9245-9ada4d29efd7\") " pod="openshift-console/console-978dfdd6c-4s4qq" Sep 11 18:18:52.128515 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:18:52.128361 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d10f5f4-8804-4de1-9245-9ada4d29efd7-console-oauth-config\") pod \"console-978dfdd6c-4s4qq\" (UID: \"1d10f5f4-8804-4de1-9245-9ada4d29efd7\") " pod="openshift-console/console-978dfdd6c-4s4qq" Sep 11 18:18:52.128515 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:18:52.128387 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d10f5f4-8804-4de1-9245-9ada4d29efd7-service-ca\") pod \"console-978dfdd6c-4s4qq\" (UID: \"1d10f5f4-8804-4de1-9245-9ada4d29efd7\") " pod="openshift-console/console-978dfdd6c-4s4qq" Sep 11 18:18:52.128515 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:18:52.128415 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d10f5f4-8804-4de1-9245-9ada4d29efd7-oauth-serving-cert\") pod \"console-978dfdd6c-4s4qq\" (UID: \"1d10f5f4-8804-4de1-9245-9ada4d29efd7\") " pod="openshift-console/console-978dfdd6c-4s4qq" Sep 11 18:18:52.128515 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:18:52.128432 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d10f5f4-8804-4de1-9245-9ada4d29efd7-console-config\") pod \"console-978dfdd6c-4s4qq\" (UID: \"1d10f5f4-8804-4de1-9245-9ada4d29efd7\") " pod="openshift-console/console-978dfdd6c-4s4qq" Sep 11 18:18:52.128694 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:18:52.128625 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d10f5f4-8804-4de1-9245-9ada4d29efd7-console-serving-cert\") pod \"console-978dfdd6c-4s4qq\" (UID: \"1d10f5f4-8804-4de1-9245-9ada4d29efd7\") " pod="openshift-console/console-978dfdd6c-4s4qq" Sep 11 18:18:52.128694 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:18:52.128663 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d10f5f4-8804-4de1-9245-9ada4d29efd7-trusted-ca-bundle\") pod \"console-978dfdd6c-4s4qq\" (UID: \"1d10f5f4-8804-4de1-9245-9ada4d29efd7\") " pod="openshift-console/console-978dfdd6c-4s4qq" Sep 11 18:18:52.129331 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:18:52.129299 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d10f5f4-8804-4de1-9245-9ada4d29efd7-service-ca\") pod \"console-978dfdd6c-4s4qq\" (UID: \"1d10f5f4-8804-4de1-9245-9ada4d29efd7\") " pod="openshift-console/console-978dfdd6c-4s4qq" Sep 11 18:18:52.129331 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:18:52.129324 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d10f5f4-8804-4de1-9245-9ada4d29efd7-console-config\") pod \"console-978dfdd6c-4s4qq\" (UID: \"1d10f5f4-8804-4de1-9245-9ada4d29efd7\") " pod="openshift-console/console-978dfdd6c-4s4qq" Sep 11 18:18:52.129331 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:18:52.129307 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d10f5f4-8804-4de1-9245-9ada4d29efd7-oauth-serving-cert\") pod \"console-978dfdd6c-4s4qq\" (UID: \"1d10f5f4-8804-4de1-9245-9ada4d29efd7\") " pod="openshift-console/console-978dfdd6c-4s4qq" Sep 11 18:18:52.129607 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:18:52.129588 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d10f5f4-8804-4de1-9245-9ada4d29efd7-trusted-ca-bundle\") pod \"console-978dfdd6c-4s4qq\" (UID: \"1d10f5f4-8804-4de1-9245-9ada4d29efd7\") " pod="openshift-console/console-978dfdd6c-4s4qq" Sep 11 18:18:52.131088 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:18:52.131043 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d10f5f4-8804-4de1-9245-9ada4d29efd7-console-oauth-config\") pod \"console-978dfdd6c-4s4qq\" (UID: \"1d10f5f4-8804-4de1-9245-9ada4d29efd7\") " pod="openshift-console/console-978dfdd6c-4s4qq" Sep 11 18:18:52.131203 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:18:52.131171 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d10f5f4-8804-4de1-9245-9ada4d29efd7-console-serving-cert\") pod \"console-978dfdd6c-4s4qq\" (UID: \"1d10f5f4-8804-4de1-9245-9ada4d29efd7\") " pod="openshift-console/console-978dfdd6c-4s4qq" Sep 11 18:18:52.137298 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:18:52.137272 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skqnr\" (UniqueName: \"kubernetes.io/projected/1d10f5f4-8804-4de1-9245-9ada4d29efd7-kube-api-access-skqnr\") pod \"console-978dfdd6c-4s4qq\" (UID: \"1d10f5f4-8804-4de1-9245-9ada4d29efd7\") " pod="openshift-console/console-978dfdd6c-4s4qq" Sep 11 18:18:52.209928 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:18:52.209893 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-978dfdd6c-4s4qq" Sep 11 18:18:52.490885 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:18:52.490857 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-978dfdd6c-4s4qq"] Sep 11 18:18:52.493555 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:18:52.493523 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d10f5f4_8804_4de1_9245_9ada4d29efd7.slice/crio-076729a011a8534a70aaf939de17a5865fad49cf95c36f1d37f81821fe085a9a WatchSource:0}: Error finding container 076729a011a8534a70aaf939de17a5865fad49cf95c36f1d37f81821fe085a9a: Status 404 returned error can't find the container with id 076729a011a8534a70aaf939de17a5865fad49cf95c36f1d37f81821fe085a9a Sep 11 18:18:53.029944 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:18:53.029907 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-978dfdd6c-4s4qq" event={"ID":"1d10f5f4-8804-4de1-9245-9ada4d29efd7","Type":"ContainerStarted","Data":"8db7742b487aed84d9dcff0b1f5f0682e8fb56b49a80708f5bedf7297e601d0d"} Sep 11 18:18:53.030369 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:18:53.029960 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-978dfdd6c-4s4qq" event={"ID":"1d10f5f4-8804-4de1-9245-9ada4d29efd7","Type":"ContainerStarted","Data":"076729a011a8534a70aaf939de17a5865fad49cf95c36f1d37f81821fe085a9a"} Sep 11 18:18:53.048422 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:18:53.048366 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-978dfdd6c-4s4qq" podStartSLOduration=2.048349506 podStartE2EDuration="2.048349506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:18:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:18:53.046641994 +0000 UTC m=+214.543659748" watchObservedRunningTime="2026-09-11 18:18:53.048349506 +0000 UTC m=+214.545367261" Sep 11 18:19:01.646593 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:01.646558 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-978dfdd6c-4s4qq"] Sep 11 18:19:01.670035 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:01.670003 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-bb5b74866-mtlnv"] Sep 11 18:19:01.674444 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:01.674415 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bb5b74866-mtlnv" Sep 11 18:19:01.686563 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:01.686532 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bb5b74866-mtlnv"] Sep 11 18:19:01.705859 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:01.705827 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5-console-config\") pod \"console-bb5b74866-mtlnv\" (UID: \"e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5\") " pod="openshift-console/console-bb5b74866-mtlnv" Sep 11 18:19:01.706039 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:01.705865 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5-service-ca\") pod \"console-bb5b74866-mtlnv\" (UID: \"e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5\") " pod="openshift-console/console-bb5b74866-mtlnv" Sep 11 18:19:01.706039 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:01.705900 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xftw\" (UniqueName: \"kubernetes.io/projected/e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5-kube-api-access-5xftw\") pod \"console-bb5b74866-mtlnv\" (UID: \"e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5\") " pod="openshift-console/console-bb5b74866-mtlnv" Sep 11 18:19:01.706039 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:01.705926 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5-oauth-serving-cert\") pod \"console-bb5b74866-mtlnv\" (UID: \"e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5\") " pod="openshift-console/console-bb5b74866-mtlnv" Sep 11 18:19:01.706039 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:01.705948 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5-console-oauth-config\") pod \"console-bb5b74866-mtlnv\" (UID: \"e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5\") " pod="openshift-console/console-bb5b74866-mtlnv" Sep 11 18:19:01.706039 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:01.705993 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5-trusted-ca-bundle\") pod \"console-bb5b74866-mtlnv\" (UID: \"e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5\") " pod="openshift-console/console-bb5b74866-mtlnv" Sep 11 18:19:01.706336 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:01.706045 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5-console-serving-cert\") pod \"console-bb5b74866-mtlnv\" (UID: \"e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5\") " pod="openshift-console/console-bb5b74866-mtlnv" Sep 11 18:19:01.806946 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:01.806903 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5-console-serving-cert\") pod \"console-bb5b74866-mtlnv\" (UID: \"e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5\") " pod="openshift-console/console-bb5b74866-mtlnv" Sep 11 18:19:01.807169 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:01.806958 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5-console-config\") pod \"console-bb5b74866-mtlnv\" (UID: \"e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5\") " pod="openshift-console/console-bb5b74866-mtlnv" Sep 11 18:19:01.807169 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:01.806988 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5-service-ca\") pod \"console-bb5b74866-mtlnv\" (UID: \"e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5\") " pod="openshift-console/console-bb5b74866-mtlnv" Sep 11 18:19:01.807169 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:01.807012 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xftw\" (UniqueName: \"kubernetes.io/projected/e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5-kube-api-access-5xftw\") pod \"console-bb5b74866-mtlnv\" (UID: \"e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5\") " pod="openshift-console/console-bb5b74866-mtlnv" Sep 11 18:19:01.807169 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:01.807035 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5-oauth-serving-cert\") pod \"console-bb5b74866-mtlnv\" (UID: \"e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5\") " pod="openshift-console/console-bb5b74866-mtlnv" Sep 11 18:19:01.807169 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:01.807085 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5-console-oauth-config\") pod \"console-bb5b74866-mtlnv\" (UID: \"e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5\") " pod="openshift-console/console-bb5b74866-mtlnv" Sep 11 18:19:01.807169 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:01.807116 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5-trusted-ca-bundle\") pod \"console-bb5b74866-mtlnv\" (UID: \"e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5\") " pod="openshift-console/console-bb5b74866-mtlnv" Sep 11 18:19:01.807860 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:01.807835 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5-service-ca\") pod \"console-bb5b74866-mtlnv\" (UID: \"e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5\") " pod="openshift-console/console-bb5b74866-mtlnv" Sep 11 18:19:01.807956 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:01.807867 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5-oauth-serving-cert\") pod \"console-bb5b74866-mtlnv\" (UID: \"e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5\") " pod="openshift-console/console-bb5b74866-mtlnv" Sep 11 18:19:01.808005 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:01.807975 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5-trusted-ca-bundle\") pod \"console-bb5b74866-mtlnv\" (UID: \"e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5\") " pod="openshift-console/console-bb5b74866-mtlnv" Sep 11 18:19:01.808230 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:01.808211 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5-console-config\") pod \"console-bb5b74866-mtlnv\" (UID: \"e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5\") " pod="openshift-console/console-bb5b74866-mtlnv" Sep 11 18:19:01.809936 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:01.809915 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5-console-serving-cert\") pod \"console-bb5b74866-mtlnv\" (UID: \"e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5\") " pod="openshift-console/console-bb5b74866-mtlnv" Sep 11 18:19:01.810012 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:01.809935 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5-console-oauth-config\") pod \"console-bb5b74866-mtlnv\" (UID: \"e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5\") " pod="openshift-console/console-bb5b74866-mtlnv" Sep 11 18:19:01.819355 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:01.819335 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xftw\" (UniqueName: \"kubernetes.io/projected/e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5-kube-api-access-5xftw\") pod \"console-bb5b74866-mtlnv\" (UID: \"e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5\") " pod="openshift-console/console-bb5b74866-mtlnv" Sep 11 18:19:01.983602 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:01.983571 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bb5b74866-mtlnv" Sep 11 18:19:02.210209 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:02.210175 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-978dfdd6c-4s4qq" Sep 11 18:19:02.257805 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:02.257723 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bb5b74866-mtlnv"] Sep 11 18:19:02.260647 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:19:02.260616 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode53d9fd4_d77d_4d4f_b12b_9d91211e5ce5.slice/crio-dd993a7fb176d91a7f32d1201d69097a978eba11d884e2409f744373795c9100 WatchSource:0}: Error finding container dd993a7fb176d91a7f32d1201d69097a978eba11d884e2409f744373795c9100: Status 404 returned error can't find the container with id dd993a7fb176d91a7f32d1201d69097a978eba11d884e2409f744373795c9100 Sep 11 18:19:03.074431 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:03.074395 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bb5b74866-mtlnv" event={"ID":"e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5","Type":"ContainerStarted","Data":"a512123646cfbeea74d4c342b5d92af674d744fa0b7e1b05af35e896ada70393"} Sep 11 18:19:03.074824 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:03.074443 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bb5b74866-mtlnv" event={"ID":"e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5","Type":"ContainerStarted","Data":"dd993a7fb176d91a7f32d1201d69097a978eba11d884e2409f744373795c9100"} Sep 11 18:19:03.092588 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:03.092525 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-bb5b74866-mtlnv" podStartSLOduration=2.092505238 podStartE2EDuration="2.092505238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:19:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:19:03.091110153 +0000 UTC m=+224.588127901" watchObservedRunningTime="2026-09-11 18:19:03.092505238 +0000 UTC m=+224.589522996" Sep 11 18:19:11.984499 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:11.984458 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-bb5b74866-mtlnv" Sep 11 18:19:11.984982 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:11.984553 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-bb5b74866-mtlnv" Sep 11 18:19:11.989280 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:11.989257 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-bb5b74866-mtlnv" Sep 11 18:19:12.113115 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:12.113086 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-bb5b74866-mtlnv" Sep 11 18:19:12.160564 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:12.160517 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-586659979-qlk7m"] Sep 11 18:19:26.668038 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:26.667973 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-978dfdd6c-4s4qq" podUID="1d10f5f4-8804-4de1-9245-9ada4d29efd7" containerName="console" containerID="cri-o://8db7742b487aed84d9dcff0b1f5f0682e8fb56b49a80708f5bedf7297e601d0d" gracePeriod=15 Sep 11 18:19:26.902953 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:26.902928 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-978dfdd6c-4s4qq" Sep 11 18:19:27.018739 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:27.018643 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d10f5f4-8804-4de1-9245-9ada4d29efd7-console-oauth-config\") pod \"1d10f5f4-8804-4de1-9245-9ada4d29efd7\" (UID: \"1d10f5f4-8804-4de1-9245-9ada4d29efd7\") " Sep 11 18:19:27.018739 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:27.018691 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d10f5f4-8804-4de1-9245-9ada4d29efd7-console-serving-cert\") pod \"1d10f5f4-8804-4de1-9245-9ada4d29efd7\" (UID: \"1d10f5f4-8804-4de1-9245-9ada4d29efd7\") " Sep 11 18:19:27.018739 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:27.018720 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d10f5f4-8804-4de1-9245-9ada4d29efd7-console-config\") pod \"1d10f5f4-8804-4de1-9245-9ada4d29efd7\" (UID: \"1d10f5f4-8804-4de1-9245-9ada4d29efd7\") " Sep 11 18:19:27.018739 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:27.018745 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d10f5f4-8804-4de1-9245-9ada4d29efd7-oauth-serving-cert\") pod \"1d10f5f4-8804-4de1-9245-9ada4d29efd7\" (UID: \"1d10f5f4-8804-4de1-9245-9ada4d29efd7\") " Sep 11 18:19:27.019093 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:27.018797 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d10f5f4-8804-4de1-9245-9ada4d29efd7-trusted-ca-bundle\") pod \"1d10f5f4-8804-4de1-9245-9ada4d29efd7\" (UID: \"1d10f5f4-8804-4de1-9245-9ada4d29efd7\") " Sep 11 18:19:27.019093 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:27.018874 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-skqnr\" (UniqueName: \"kubernetes.io/projected/1d10f5f4-8804-4de1-9245-9ada4d29efd7-kube-api-access-skqnr\") pod \"1d10f5f4-8804-4de1-9245-9ada4d29efd7\" (UID: \"1d10f5f4-8804-4de1-9245-9ada4d29efd7\") " Sep 11 18:19:27.019093 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:27.018919 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d10f5f4-8804-4de1-9245-9ada4d29efd7-service-ca\") pod \"1d10f5f4-8804-4de1-9245-9ada4d29efd7\" (UID: \"1d10f5f4-8804-4de1-9245-9ada4d29efd7\") " Sep 11 18:19:27.019358 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:27.019327 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d10f5f4-8804-4de1-9245-9ada4d29efd7-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1d10f5f4-8804-4de1-9245-9ada4d29efd7" (UID: "1d10f5f4-8804-4de1-9245-9ada4d29efd7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:19:27.019428 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:27.019333 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d10f5f4-8804-4de1-9245-9ada4d29efd7-console-config" (OuterVolumeSpecName: "console-config") pod "1d10f5f4-8804-4de1-9245-9ada4d29efd7" (UID: "1d10f5f4-8804-4de1-9245-9ada4d29efd7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:19:27.019594 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:27.019564 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d10f5f4-8804-4de1-9245-9ada4d29efd7-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "1d10f5f4-8804-4de1-9245-9ada4d29efd7" (UID: "1d10f5f4-8804-4de1-9245-9ada4d29efd7"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:19:27.019677 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:27.019600 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d10f5f4-8804-4de1-9245-9ada4d29efd7-service-ca" (OuterVolumeSpecName: "service-ca") pod "1d10f5f4-8804-4de1-9245-9ada4d29efd7" (UID: "1d10f5f4-8804-4de1-9245-9ada4d29efd7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:19:27.019677 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:27.019638 2561 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d10f5f4-8804-4de1-9245-9ada4d29efd7-console-config\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:19:27.019677 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:27.019654 2561 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d10f5f4-8804-4de1-9245-9ada4d29efd7-oauth-serving-cert\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:19:27.021110 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:27.021084 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d10f5f4-8804-4de1-9245-9ada4d29efd7-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1d10f5f4-8804-4de1-9245-9ada4d29efd7" (UID: "1d10f5f4-8804-4de1-9245-9ada4d29efd7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:19:27.021228 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:27.021118 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d10f5f4-8804-4de1-9245-9ada4d29efd7-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1d10f5f4-8804-4de1-9245-9ada4d29efd7" (UID: "1d10f5f4-8804-4de1-9245-9ada4d29efd7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:19:27.021228 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:27.021199 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d10f5f4-8804-4de1-9245-9ada4d29efd7-kube-api-access-skqnr" (OuterVolumeSpecName: "kube-api-access-skqnr") pod "1d10f5f4-8804-4de1-9245-9ada4d29efd7" (UID: "1d10f5f4-8804-4de1-9245-9ada4d29efd7"). InnerVolumeSpecName "kube-api-access-skqnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:19:27.120142 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:27.120098 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skqnr\" (UniqueName: \"kubernetes.io/projected/1d10f5f4-8804-4de1-9245-9ada4d29efd7-kube-api-access-skqnr\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:19:27.120142 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:27.120136 2561 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d10f5f4-8804-4de1-9245-9ada4d29efd7-service-ca\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:19:27.120142 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:27.120151 2561 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d10f5f4-8804-4de1-9245-9ada4d29efd7-console-oauth-config\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:19:27.120375 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:27.120163 2561 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d10f5f4-8804-4de1-9245-9ada4d29efd7-console-serving-cert\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:19:27.120375 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:27.120174 2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d10f5f4-8804-4de1-9245-9ada4d29efd7-trusted-ca-bundle\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:19:27.167990 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:27.167954 2561 generic.go:356] "Generic (PLEG): container finished" podID="1d10f5f4-8804-4de1-9245-9ada4d29efd7" containerID="8db7742b487aed84d9dcff0b1f5f0682e8fb56b49a80708f5bedf7297e601d0d" exitCode=0 Sep 11 18:19:27.167990 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:27.168001 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-978dfdd6c-4s4qq" Sep 11 18:19:27.168220 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:27.168039 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-978dfdd6c-4s4qq" event={"ID":"1d10f5f4-8804-4de1-9245-9ada4d29efd7","Type":"ContainerDied","Data":"8db7742b487aed84d9dcff0b1f5f0682e8fb56b49a80708f5bedf7297e601d0d"} Sep 11 18:19:27.168220 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:27.168105 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-978dfdd6c-4s4qq" event={"ID":"1d10f5f4-8804-4de1-9245-9ada4d29efd7","Type":"ContainerDied","Data":"076729a011a8534a70aaf939de17a5865fad49cf95c36f1d37f81821fe085a9a"} Sep 11 18:19:27.168220 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:27.168125 2561 scope.go:117] "RemoveContainer" containerID="8db7742b487aed84d9dcff0b1f5f0682e8fb56b49a80708f5bedf7297e601d0d" Sep 11 18:19:27.176457 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:27.176431 2561 scope.go:117] "RemoveContainer" containerID="8db7742b487aed84d9dcff0b1f5f0682e8fb56b49a80708f5bedf7297e601d0d" Sep 11 18:19:27.176748 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:19:27.176731 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8db7742b487aed84d9dcff0b1f5f0682e8fb56b49a80708f5bedf7297e601d0d\": container with ID starting with 8db7742b487aed84d9dcff0b1f5f0682e8fb56b49a80708f5bedf7297e601d0d not found: ID does not exist" containerID="8db7742b487aed84d9dcff0b1f5f0682e8fb56b49a80708f5bedf7297e601d0d" Sep 11 18:19:27.176815 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:27.176758 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8db7742b487aed84d9dcff0b1f5f0682e8fb56b49a80708f5bedf7297e601d0d"} err="failed to get container status \"8db7742b487aed84d9dcff0b1f5f0682e8fb56b49a80708f5bedf7297e601d0d\": rpc error: code = NotFound desc = could not find container \"8db7742b487aed84d9dcff0b1f5f0682e8fb56b49a80708f5bedf7297e601d0d\": container with ID starting with 8db7742b487aed84d9dcff0b1f5f0682e8fb56b49a80708f5bedf7297e601d0d not found: ID does not exist" Sep 11 18:19:27.186581 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:27.186553 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-978dfdd6c-4s4qq"] Sep 11 18:19:27.188884 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:27.188860 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-978dfdd6c-4s4qq"] Sep 11 18:19:29.137468 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:29.137431 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d10f5f4-8804-4de1-9245-9ada4d29efd7" path="/var/lib/kubelet/pods/1d10f5f4-8804-4de1-9245-9ada4d29efd7/volumes" Sep 11 18:19:37.180576 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:37.180500 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-586659979-qlk7m" podUID="6f85e5c5-b612-4d6e-ac5a-4dca63a8980d" containerName="console" containerID="cri-o://2c4cfb6f91e58433bf0a400095ba1ba51da509266e8771c5f3e29087753cc3b9" gracePeriod=15 Sep 11 18:19:37.431132 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:37.431073 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-586659979-qlk7m" Sep 11 18:19:37.511609 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:37.511572 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6f85e5c5-b612-4d6e-ac5a-4dca63a8980d-service-ca\") pod \"6f85e5c5-b612-4d6e-ac5a-4dca63a8980d\" (UID: \"6f85e5c5-b612-4d6e-ac5a-4dca63a8980d\") " Sep 11 18:19:37.511609 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:37.511614 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6f85e5c5-b612-4d6e-ac5a-4dca63a8980d-oauth-serving-cert\") pod \"6f85e5c5-b612-4d6e-ac5a-4dca63a8980d\" (UID: \"6f85e5c5-b612-4d6e-ac5a-4dca63a8980d\") " Sep 11 18:19:37.511842 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:37.511656 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6f85e5c5-b612-4d6e-ac5a-4dca63a8980d-console-config\") pod \"6f85e5c5-b612-4d6e-ac5a-4dca63a8980d\" (UID: \"6f85e5c5-b612-4d6e-ac5a-4dca63a8980d\") " Sep 11 18:19:37.511842 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:37.511691 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f85e5c5-b612-4d6e-ac5a-4dca63a8980d-trusted-ca-bundle\") pod \"6f85e5c5-b612-4d6e-ac5a-4dca63a8980d\" (UID: \"6f85e5c5-b612-4d6e-ac5a-4dca63a8980d\") " Sep 11 18:19:37.511842 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:37.511747 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6f85e5c5-b612-4d6e-ac5a-4dca63a8980d-console-serving-cert\") pod \"6f85e5c5-b612-4d6e-ac5a-4dca63a8980d\" (UID: \"6f85e5c5-b612-4d6e-ac5a-4dca63a8980d\") " Sep 11 18:19:37.511842 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:37.511779 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6f85e5c5-b612-4d6e-ac5a-4dca63a8980d-console-oauth-config\") pod \"6f85e5c5-b612-4d6e-ac5a-4dca63a8980d\" (UID: \"6f85e5c5-b612-4d6e-ac5a-4dca63a8980d\") " Sep 11 18:19:37.511842 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:37.511822 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q98qx\" (UniqueName: \"kubernetes.io/projected/6f85e5c5-b612-4d6e-ac5a-4dca63a8980d-kube-api-access-q98qx\") pod \"6f85e5c5-b612-4d6e-ac5a-4dca63a8980d\" (UID: \"6f85e5c5-b612-4d6e-ac5a-4dca63a8980d\") " Sep 11 18:19:37.512163 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:37.512130 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f85e5c5-b612-4d6e-ac5a-4dca63a8980d-service-ca" (OuterVolumeSpecName: "service-ca") pod "6f85e5c5-b612-4d6e-ac5a-4dca63a8980d" (UID: "6f85e5c5-b612-4d6e-ac5a-4dca63a8980d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:19:37.512254 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:37.512163 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f85e5c5-b612-4d6e-ac5a-4dca63a8980d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6f85e5c5-b612-4d6e-ac5a-4dca63a8980d" (UID: "6f85e5c5-b612-4d6e-ac5a-4dca63a8980d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:19:37.512254 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:37.512229 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f85e5c5-b612-4d6e-ac5a-4dca63a8980d-console-config" (OuterVolumeSpecName: "console-config") pod "6f85e5c5-b612-4d6e-ac5a-4dca63a8980d" (UID: "6f85e5c5-b612-4d6e-ac5a-4dca63a8980d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:19:37.512336 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:37.512270 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f85e5c5-b612-4d6e-ac5a-4dca63a8980d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "6f85e5c5-b612-4d6e-ac5a-4dca63a8980d" (UID: "6f85e5c5-b612-4d6e-ac5a-4dca63a8980d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:19:37.514068 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:37.514028 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f85e5c5-b612-4d6e-ac5a-4dca63a8980d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6f85e5c5-b612-4d6e-ac5a-4dca63a8980d" (UID: "6f85e5c5-b612-4d6e-ac5a-4dca63a8980d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:19:37.514149 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:37.514082 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f85e5c5-b612-4d6e-ac5a-4dca63a8980d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6f85e5c5-b612-4d6e-ac5a-4dca63a8980d" (UID: "6f85e5c5-b612-4d6e-ac5a-4dca63a8980d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:19:37.514225 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:37.514203 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f85e5c5-b612-4d6e-ac5a-4dca63a8980d-kube-api-access-q98qx" (OuterVolumeSpecName: "kube-api-access-q98qx") pod "6f85e5c5-b612-4d6e-ac5a-4dca63a8980d" (UID: "6f85e5c5-b612-4d6e-ac5a-4dca63a8980d"). InnerVolumeSpecName "kube-api-access-q98qx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:19:37.612839 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:37.612797 2561 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6f85e5c5-b612-4d6e-ac5a-4dca63a8980d-service-ca\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:19:37.612839 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:37.612833 2561 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6f85e5c5-b612-4d6e-ac5a-4dca63a8980d-oauth-serving-cert\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:19:37.613039 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:37.612846 2561 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6f85e5c5-b612-4d6e-ac5a-4dca63a8980d-console-config\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:19:37.613039 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:37.612860 2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f85e5c5-b612-4d6e-ac5a-4dca63a8980d-trusted-ca-bundle\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:19:37.613039 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:37.612872 2561 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6f85e5c5-b612-4d6e-ac5a-4dca63a8980d-console-serving-cert\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:19:37.613039 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:37.612883 2561 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6f85e5c5-b612-4d6e-ac5a-4dca63a8980d-console-oauth-config\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:19:37.613039 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:37.612894 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q98qx\" (UniqueName: \"kubernetes.io/projected/6f85e5c5-b612-4d6e-ac5a-4dca63a8980d-kube-api-access-q98qx\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:19:38.210830 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:38.210796 2561 generic.go:356] "Generic (PLEG): container finished" podID="6f85e5c5-b612-4d6e-ac5a-4dca63a8980d" containerID="2c4cfb6f91e58433bf0a400095ba1ba51da509266e8771c5f3e29087753cc3b9" exitCode=0 Sep 11 18:19:38.211264 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:38.210836 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-586659979-qlk7m" Sep 11 18:19:38.211264 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:38.210866 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-586659979-qlk7m" event={"ID":"6f85e5c5-b612-4d6e-ac5a-4dca63a8980d","Type":"ContainerDied","Data":"2c4cfb6f91e58433bf0a400095ba1ba51da509266e8771c5f3e29087753cc3b9"} Sep 11 18:19:38.211264 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:38.210905 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-586659979-qlk7m" event={"ID":"6f85e5c5-b612-4d6e-ac5a-4dca63a8980d","Type":"ContainerDied","Data":"f65e19056f87a007f0abdcc4d209891ae53d87094198b3c6bdc377210832a0b4"} Sep 11 18:19:38.211264 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:38.210925 2561 scope.go:117] "RemoveContainer" containerID="2c4cfb6f91e58433bf0a400095ba1ba51da509266e8771c5f3e29087753cc3b9" Sep 11 18:19:38.219746 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:38.219726 2561 scope.go:117] "RemoveContainer" containerID="2c4cfb6f91e58433bf0a400095ba1ba51da509266e8771c5f3e29087753cc3b9" Sep 11 18:19:38.220042 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:19:38.220022 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c4cfb6f91e58433bf0a400095ba1ba51da509266e8771c5f3e29087753cc3b9\": container with ID starting with 2c4cfb6f91e58433bf0a400095ba1ba51da509266e8771c5f3e29087753cc3b9 not found: ID does not exist" containerID="2c4cfb6f91e58433bf0a400095ba1ba51da509266e8771c5f3e29087753cc3b9" Sep 11 18:19:38.220105 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:38.220063 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c4cfb6f91e58433bf0a400095ba1ba51da509266e8771c5f3e29087753cc3b9"} err="failed to get container status \"2c4cfb6f91e58433bf0a400095ba1ba51da509266e8771c5f3e29087753cc3b9\": rpc error: code = NotFound desc = could not find container \"2c4cfb6f91e58433bf0a400095ba1ba51da509266e8771c5f3e29087753cc3b9\": container with ID starting with 2c4cfb6f91e58433bf0a400095ba1ba51da509266e8771c5f3e29087753cc3b9 not found: ID does not exist" Sep 11 18:19:38.233399 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:38.233369 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-586659979-qlk7m"] Sep 11 18:19:38.238587 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:38.238563 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-586659979-qlk7m"] Sep 11 18:19:39.137934 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:19:39.137898 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f85e5c5-b612-4d6e-ac5a-4dca63a8980d" path="/var/lib/kubelet/pods/6f85e5c5-b612-4d6e-ac5a-4dca63a8980d/volumes" Sep 11 18:20:11.558926 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:11.558892 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540945t6"] Sep 11 18:20:11.559509 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:11.559296 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f85e5c5-b612-4d6e-ac5a-4dca63a8980d" containerName="console" Sep 11 18:20:11.559509 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:11.559315 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f85e5c5-b612-4d6e-ac5a-4dca63a8980d" containerName="console" Sep 11 18:20:11.559509 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:11.559350 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d10f5f4-8804-4de1-9245-9ada4d29efd7" containerName="console" Sep 11 18:20:11.559509 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:11.559379 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d10f5f4-8804-4de1-9245-9ada4d29efd7" containerName="console" Sep 11 18:20:11.559509 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:11.559448 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d10f5f4-8804-4de1-9245-9ada4d29efd7" containerName="console" Sep 11 18:20:11.559509 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:11.559459 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f85e5c5-b612-4d6e-ac5a-4dca63a8980d" containerName="console" Sep 11 18:20:11.562550 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:11.562528 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540945t6" Sep 11 18:20:11.564972 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:11.564950 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-gfgp7\"" Sep 11 18:20:11.565107 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:11.564951 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 11 18:20:11.565842 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:11.565826 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 11 18:20:11.570958 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:11.570935 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540945t6"] Sep 11 18:20:11.572474 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:11.572456 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ff070c4-d29e-473f-83b2-c9ff022cb89f-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540945t6\" (UID: \"8ff070c4-d29e-473f-83b2-c9ff022cb89f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540945t6" Sep 11 18:20:11.572560 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:11.572503 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ff070c4-d29e-473f-83b2-c9ff022cb89f-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540945t6\" (UID: \"8ff070c4-d29e-473f-83b2-c9ff022cb89f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540945t6" Sep 11 18:20:11.572601 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:11.572572 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmk5c\" (UniqueName: \"kubernetes.io/projected/8ff070c4-d29e-473f-83b2-c9ff022cb89f-kube-api-access-mmk5c\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540945t6\" (UID: \"8ff070c4-d29e-473f-83b2-c9ff022cb89f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540945t6" Sep 11 18:20:11.673186 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:11.673149 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmk5c\" (UniqueName: \"kubernetes.io/projected/8ff070c4-d29e-473f-83b2-c9ff022cb89f-kube-api-access-mmk5c\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540945t6\" (UID: \"8ff070c4-d29e-473f-83b2-c9ff022cb89f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540945t6" Sep 11 18:20:11.673186 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:11.673191 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ff070c4-d29e-473f-83b2-c9ff022cb89f-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540945t6\" (UID: \"8ff070c4-d29e-473f-83b2-c9ff022cb89f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540945t6" Sep 11 18:20:11.673444 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:11.673244 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ff070c4-d29e-473f-83b2-c9ff022cb89f-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540945t6\" (UID: \"8ff070c4-d29e-473f-83b2-c9ff022cb89f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540945t6" Sep 11 18:20:11.673669 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:11.673648 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ff070c4-d29e-473f-83b2-c9ff022cb89f-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540945t6\" (UID: \"8ff070c4-d29e-473f-83b2-c9ff022cb89f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540945t6" Sep 11 18:20:11.673722 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:11.673658 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ff070c4-d29e-473f-83b2-c9ff022cb89f-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540945t6\" (UID: \"8ff070c4-d29e-473f-83b2-c9ff022cb89f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540945t6" Sep 11 18:20:11.682527 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:11.682496 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmk5c\" (UniqueName: \"kubernetes.io/projected/8ff070c4-d29e-473f-83b2-c9ff022cb89f-kube-api-access-mmk5c\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540945t6\" (UID: \"8ff070c4-d29e-473f-83b2-c9ff022cb89f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540945t6" Sep 11 18:20:11.871788 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:11.871702 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540945t6" Sep 11 18:20:12.141647 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:12.141556 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540945t6"] Sep 11 18:20:12.143764 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:20:12.143739 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ff070c4_d29e_473f_83b2_c9ff022cb89f.slice/crio-bc7448af49886b6969f9f5018173313abe6cc4aad6f17a766c846ac468b7937e WatchSource:0}: Error finding container bc7448af49886b6969f9f5018173313abe6cc4aad6f17a766c846ac468b7937e: Status 404 returned error can't find the container with id bc7448af49886b6969f9f5018173313abe6cc4aad6f17a766c846ac468b7937e Sep 11 18:20:12.339157 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:12.339124 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540945t6" event={"ID":"8ff070c4-d29e-473f-83b2-c9ff022cb89f","Type":"ContainerStarted","Data":"bc7448af49886b6969f9f5018173313abe6cc4aad6f17a766c846ac468b7937e"} Sep 11 18:20:19.236470 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:19.236439 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dj524_60702ef3-98ef-463b-b877-3a00825353d8/console-operator/1.log" Sep 11 18:20:19.236921 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:19.236903 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dj524_60702ef3-98ef-463b-b877-3a00825353d8/console-operator/1.log" Sep 11 18:20:19.243910 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:19.243883 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4knd_42e5ddd8-a96c-45a1-adf9-caa47ee902db/ovn-acl-logging/0.log" Sep 11 18:20:19.244215 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:19.244196 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4knd_42e5ddd8-a96c-45a1-adf9-caa47ee902db/ovn-acl-logging/0.log" Sep 11 18:20:19.247530 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:19.247512 2561 kubelet.go:1648] "Image garbage collection succeeded" Sep 11 18:20:19.373200 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:19.368124 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540945t6" event={"ID":"8ff070c4-d29e-473f-83b2-c9ff022cb89f","Type":"ContainerStarted","Data":"d7492026d8fb4922289020917a2c9dce47d81823dd366304e73e9eca422b965c"} Sep 11 18:20:20.373623 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:20.373589 2561 generic.go:356] "Generic (PLEG): container finished" podID="8ff070c4-d29e-473f-83b2-c9ff022cb89f" containerID="d7492026d8fb4922289020917a2c9dce47d81823dd366304e73e9eca422b965c" exitCode=0 Sep 11 18:20:20.374015 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:20.373653 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540945t6" event={"ID":"8ff070c4-d29e-473f-83b2-c9ff022cb89f","Type":"ContainerDied","Data":"d7492026d8fb4922289020917a2c9dce47d81823dd366304e73e9eca422b965c"} Sep 11 18:20:20.374839 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:20.374823 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 18:20:23.386842 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:23.386803 2561 generic.go:356] "Generic (PLEG): container finished" podID="8ff070c4-d29e-473f-83b2-c9ff022cb89f" containerID="93992ece921e896bd3de2989577c7371bf26cd9a14fc2c963508c1837e0c60c9" exitCode=0 Sep 11 18:20:23.387305 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:23.386882 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540945t6" event={"ID":"8ff070c4-d29e-473f-83b2-c9ff022cb89f","Type":"ContainerDied","Data":"93992ece921e896bd3de2989577c7371bf26cd9a14fc2c963508c1837e0c60c9"} Sep 11 18:20:33.432976 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:33.432936 2561 generic.go:356] "Generic (PLEG): container finished" podID="8ff070c4-d29e-473f-83b2-c9ff022cb89f" containerID="7653b6887a8087c0b4b066d42631a46bb623b3e216e6d8a9f5a931d29ed97279" exitCode=0 Sep 11 18:20:33.433414 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:33.433021 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540945t6" event={"ID":"8ff070c4-d29e-473f-83b2-c9ff022cb89f","Type":"ContainerDied","Data":"7653b6887a8087c0b4b066d42631a46bb623b3e216e6d8a9f5a931d29ed97279"} Sep 11 18:20:34.604593 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:34.604554 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540945t6" Sep 11 18:20:34.672650 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:34.672610 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ff070c4-d29e-473f-83b2-c9ff022cb89f-bundle\") pod \"8ff070c4-d29e-473f-83b2-c9ff022cb89f\" (UID: \"8ff070c4-d29e-473f-83b2-c9ff022cb89f\") " Sep 11 18:20:34.672650 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:34.672652 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ff070c4-d29e-473f-83b2-c9ff022cb89f-util\") pod \"8ff070c4-d29e-473f-83b2-c9ff022cb89f\" (UID: \"8ff070c4-d29e-473f-83b2-c9ff022cb89f\") " Sep 11 18:20:34.672867 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:34.672670 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmk5c\" (UniqueName: \"kubernetes.io/projected/8ff070c4-d29e-473f-83b2-c9ff022cb89f-kube-api-access-mmk5c\") pod \"8ff070c4-d29e-473f-83b2-c9ff022cb89f\" (UID: \"8ff070c4-d29e-473f-83b2-c9ff022cb89f\") " Sep 11 18:20:34.673432 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:34.673407 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ff070c4-d29e-473f-83b2-c9ff022cb89f-bundle" (OuterVolumeSpecName: "bundle") pod "8ff070c4-d29e-473f-83b2-c9ff022cb89f" (UID: "8ff070c4-d29e-473f-83b2-c9ff022cb89f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:20:34.674907 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:34.674881 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ff070c4-d29e-473f-83b2-c9ff022cb89f-kube-api-access-mmk5c" (OuterVolumeSpecName: "kube-api-access-mmk5c") pod "8ff070c4-d29e-473f-83b2-c9ff022cb89f" (UID: "8ff070c4-d29e-473f-83b2-c9ff022cb89f"). InnerVolumeSpecName "kube-api-access-mmk5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:20:34.677388 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:34.677364 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ff070c4-d29e-473f-83b2-c9ff022cb89f-util" (OuterVolumeSpecName: "util") pod "8ff070c4-d29e-473f-83b2-c9ff022cb89f" (UID: "8ff070c4-d29e-473f-83b2-c9ff022cb89f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:20:34.774158 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:34.774038 2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ff070c4-d29e-473f-83b2-c9ff022cb89f-util\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:20:34.774158 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:34.774097 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmk5c\" (UniqueName: \"kubernetes.io/projected/8ff070c4-d29e-473f-83b2-c9ff022cb89f-kube-api-access-mmk5c\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:20:34.774158 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:34.774131 2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ff070c4-d29e-473f-83b2-c9ff022cb89f-bundle\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:20:35.443192 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:35.443159 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540945t6" event={"ID":"8ff070c4-d29e-473f-83b2-c9ff022cb89f","Type":"ContainerDied","Data":"bc7448af49886b6969f9f5018173313abe6cc4aad6f17a766c846ac468b7937e"} Sep 11 18:20:35.443192 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:35.443179 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540945t6" Sep 11 18:20:35.443388 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:35.443205 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc7448af49886b6969f9f5018173313abe6cc4aad6f17a766c846ac468b7937e" Sep 11 18:20:39.371624 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:39.371590 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2pnn9"] Sep 11 18:20:39.372094 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:39.371855 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ff070c4-d29e-473f-83b2-c9ff022cb89f" containerName="pull" Sep 11 18:20:39.372094 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:39.371866 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ff070c4-d29e-473f-83b2-c9ff022cb89f" containerName="pull" Sep 11 18:20:39.372094 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:39.371883 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ff070c4-d29e-473f-83b2-c9ff022cb89f" containerName="util" Sep 11 18:20:39.372094 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:39.371888 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ff070c4-d29e-473f-83b2-c9ff022cb89f" containerName="util" Sep 11 18:20:39.372094 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:39.371902 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ff070c4-d29e-473f-83b2-c9ff022cb89f" containerName="extract" Sep 11 18:20:39.372094 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:39.371908 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ff070c4-d29e-473f-83b2-c9ff022cb89f" containerName="extract" Sep 11 18:20:39.372094 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:39.371952 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ff070c4-d29e-473f-83b2-c9ff022cb89f" containerName="extract" Sep 11 18:20:39.373752 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:39.373734 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2pnn9" Sep 11 18:20:39.376390 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:39.376366 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Sep 11 18:20:39.376572 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:39.376555 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Sep 11 18:20:39.376645 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:39.376589 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-xqn2p\"" Sep 11 18:20:39.390862 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:39.390832 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2pnn9"] Sep 11 18:20:39.518382 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:39.518342 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/716c9473-e04e-4e6c-972a-0ac6af7b9020-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-2pnn9\" (UID: \"716c9473-e04e-4e6c-972a-0ac6af7b9020\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2pnn9" Sep 11 18:20:39.518567 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:39.518414 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6xrk\" (UniqueName: \"kubernetes.io/projected/716c9473-e04e-4e6c-972a-0ac6af7b9020-kube-api-access-p6xrk\") pod \"cert-manager-operator-controller-manager-55c444bf86-2pnn9\" (UID: \"716c9473-e04e-4e6c-972a-0ac6af7b9020\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2pnn9" Sep 11 18:20:39.619426 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:39.619369 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/716c9473-e04e-4e6c-972a-0ac6af7b9020-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-2pnn9\" (UID: \"716c9473-e04e-4e6c-972a-0ac6af7b9020\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2pnn9" Sep 11 18:20:39.619598 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:39.619462 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6xrk\" (UniqueName: \"kubernetes.io/projected/716c9473-e04e-4e6c-972a-0ac6af7b9020-kube-api-access-p6xrk\") pod \"cert-manager-operator-controller-manager-55c444bf86-2pnn9\" (UID: \"716c9473-e04e-4e6c-972a-0ac6af7b9020\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2pnn9" Sep 11 18:20:39.619806 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:39.619787 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/716c9473-e04e-4e6c-972a-0ac6af7b9020-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-2pnn9\" (UID: \"716c9473-e04e-4e6c-972a-0ac6af7b9020\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2pnn9" Sep 11 18:20:39.631974 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:39.631905 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6xrk\" (UniqueName: \"kubernetes.io/projected/716c9473-e04e-4e6c-972a-0ac6af7b9020-kube-api-access-p6xrk\") pod \"cert-manager-operator-controller-manager-55c444bf86-2pnn9\" (UID: \"716c9473-e04e-4e6c-972a-0ac6af7b9020\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2pnn9" Sep 11 18:20:39.683011 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:39.682746 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2pnn9" Sep 11 18:20:39.970881 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:39.970812 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2pnn9"] Sep 11 18:20:39.974110 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:20:39.974081 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod716c9473_e04e_4e6c_972a_0ac6af7b9020.slice/crio-0039d41de8f4438b7b4c46f52e5c36d25ec3dea135f91b5101e221e068382dfd WatchSource:0}: Error finding container 0039d41de8f4438b7b4c46f52e5c36d25ec3dea135f91b5101e221e068382dfd: Status 404 returned error can't find the container with id 0039d41de8f4438b7b4c46f52e5c36d25ec3dea135f91b5101e221e068382dfd Sep 11 18:20:40.463484 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:40.463444 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2pnn9" event={"ID":"716c9473-e04e-4e6c-972a-0ac6af7b9020","Type":"ContainerStarted","Data":"0039d41de8f4438b7b4c46f52e5c36d25ec3dea135f91b5101e221e068382dfd"} Sep 11 18:20:42.474680 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:42.474608 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2pnn9" event={"ID":"716c9473-e04e-4e6c-972a-0ac6af7b9020","Type":"ContainerStarted","Data":"98e5a15fa5126a97fdf58f7f8d441b9e393d68a68e16f4fd1b810e9df436e3a9"} Sep 11 18:20:42.508473 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:42.508413 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2pnn9" podStartSLOduration=1.8659659560000001 podStartE2EDuration="3.508396625s" totalImagesPullingTime="1.642430669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:20:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:20:42.506334958 +0000 UTC m=+324.003352711" watchObservedRunningTime="2026-09-11 18:20:42.508396625 +0000 UTC m=+324.005414378" Sep 11 18:20:43.527997 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:43.527964 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-4tfgc"] Sep 11 18:20:43.530530 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:43.530507 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-4tfgc" Sep 11 18:20:43.532873 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:43.532850 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 11 18:20:43.533781 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:43.533762 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-vx28q\"" Sep 11 18:20:43.533860 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:43.533762 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 11 18:20:43.541929 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:43.541907 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-4tfgc"] Sep 11 18:20:43.652415 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:43.652372 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx25m\" (UniqueName: \"kubernetes.io/projected/85438467-07eb-41d8-94c9-8dd0fd0f2975-kube-api-access-qx25m\") pod \"cert-manager-webhook-6486ffd997-4tfgc\" (UID: \"85438467-07eb-41d8-94c9-8dd0fd0f2975\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4tfgc" Sep 11 18:20:43.652576 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:43.652432 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/85438467-07eb-41d8-94c9-8dd0fd0f2975-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-4tfgc\" (UID: \"85438467-07eb-41d8-94c9-8dd0fd0f2975\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4tfgc" Sep 11 18:20:43.753383 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:43.753344 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/85438467-07eb-41d8-94c9-8dd0fd0f2975-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-4tfgc\" (UID: \"85438467-07eb-41d8-94c9-8dd0fd0f2975\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4tfgc" Sep 11 18:20:43.753582 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:43.753436 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx25m\" (UniqueName: \"kubernetes.io/projected/85438467-07eb-41d8-94c9-8dd0fd0f2975-kube-api-access-qx25m\") pod \"cert-manager-webhook-6486ffd997-4tfgc\" (UID: \"85438467-07eb-41d8-94c9-8dd0fd0f2975\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4tfgc" Sep 11 18:20:43.762567 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:43.762527 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/85438467-07eb-41d8-94c9-8dd0fd0f2975-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-4tfgc\" (UID: \"85438467-07eb-41d8-94c9-8dd0fd0f2975\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4tfgc" Sep 11 18:20:43.762765 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:43.762746 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx25m\" (UniqueName: \"kubernetes.io/projected/85438467-07eb-41d8-94c9-8dd0fd0f2975-kube-api-access-qx25m\") pod \"cert-manager-webhook-6486ffd997-4tfgc\" (UID: \"85438467-07eb-41d8-94c9-8dd0fd0f2975\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4tfgc" Sep 11 18:20:43.839227 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:43.839141 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-4tfgc" Sep 11 18:20:44.049402 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:44.049367 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgs7td"] Sep 11 18:20:44.051570 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:44.051550 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgs7td" Sep 11 18:20:44.053880 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:44.053850 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 11 18:20:44.054022 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:44.053887 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 11 18:20:44.054022 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:44.053959 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-gfgp7\"" Sep 11 18:20:44.061582 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:44.061552 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgs7td"] Sep 11 18:20:44.109368 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:44.109340 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-4tfgc"] Sep 11 18:20:44.111967 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:20:44.111938 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod85438467_07eb_41d8_94c9_8dd0fd0f2975.slice/crio-c5c78ecdacdc08d3e75947a77c09f9f518f66e5aab6ff889990b62c6ec3a96e8 WatchSource:0}: Error finding container c5c78ecdacdc08d3e75947a77c09f9f518f66e5aab6ff889990b62c6ec3a96e8: Status 404 returned error can't find the container with id c5c78ecdacdc08d3e75947a77c09f9f518f66e5aab6ff889990b62c6ec3a96e8 Sep 11 18:20:44.156738 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:44.156705 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqb45\" (UniqueName: \"kubernetes.io/projected/63719be4-37eb-4560-a440-44894947f8c6-kube-api-access-wqb45\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgs7td\" (UID: \"63719be4-37eb-4560-a440-44894947f8c6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgs7td" Sep 11 18:20:44.156912 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:44.156772 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/63719be4-37eb-4560-a440-44894947f8c6-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgs7td\" (UID: \"63719be4-37eb-4560-a440-44894947f8c6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgs7td" Sep 11 18:20:44.156912 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:44.156800 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/63719be4-37eb-4560-a440-44894947f8c6-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgs7td\" (UID: \"63719be4-37eb-4560-a440-44894947f8c6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgs7td" Sep 11 18:20:44.257236 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:44.257197 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqb45\" (UniqueName: \"kubernetes.io/projected/63719be4-37eb-4560-a440-44894947f8c6-kube-api-access-wqb45\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgs7td\" (UID: \"63719be4-37eb-4560-a440-44894947f8c6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgs7td" Sep 11 18:20:44.257430 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:44.257315 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/63719be4-37eb-4560-a440-44894947f8c6-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgs7td\" (UID: \"63719be4-37eb-4560-a440-44894947f8c6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgs7td" Sep 11 18:20:44.257430 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:44.257351 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/63719be4-37eb-4560-a440-44894947f8c6-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgs7td\" (UID: \"63719be4-37eb-4560-a440-44894947f8c6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgs7td" Sep 11 18:20:44.257734 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:44.257709 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/63719be4-37eb-4560-a440-44894947f8c6-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgs7td\" (UID: \"63719be4-37eb-4560-a440-44894947f8c6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgs7td" Sep 11 18:20:44.257782 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:44.257717 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/63719be4-37eb-4560-a440-44894947f8c6-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgs7td\" (UID: \"63719be4-37eb-4560-a440-44894947f8c6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgs7td" Sep 11 18:20:44.267220 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:44.267190 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqb45\" (UniqueName: \"kubernetes.io/projected/63719be4-37eb-4560-a440-44894947f8c6-kube-api-access-wqb45\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgs7td\" (UID: \"63719be4-37eb-4560-a440-44894947f8c6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgs7td" Sep 11 18:20:44.362378 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:44.362283 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgs7td" Sep 11 18:20:44.483876 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:44.483840 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-4tfgc" event={"ID":"85438467-07eb-41d8-94c9-8dd0fd0f2975","Type":"ContainerStarted","Data":"c5c78ecdacdc08d3e75947a77c09f9f518f66e5aab6ff889990b62c6ec3a96e8"} Sep 11 18:20:44.633718 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:44.633685 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgs7td"] Sep 11 18:20:44.635921 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:20:44.635893 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63719be4_37eb_4560_a440_44894947f8c6.slice/crio-e01506f1811543fe5cce3540259249c7c2d965ea4b38cbe418419f2b3c2e37f2 WatchSource:0}: Error finding container e01506f1811543fe5cce3540259249c7c2d965ea4b38cbe418419f2b3c2e37f2: Status 404 returned error can't find the container with id e01506f1811543fe5cce3540259249c7c2d965ea4b38cbe418419f2b3c2e37f2 Sep 11 18:20:45.490108 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:45.490043 2561 generic.go:356] "Generic (PLEG): container finished" podID="63719be4-37eb-4560-a440-44894947f8c6" containerID="0bb81733d9e3c946826d726cb92877482626616f648c0920e3db44e84022cf7b" exitCode=0 Sep 11 18:20:45.490108 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:45.490098 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgs7td" event={"ID":"63719be4-37eb-4560-a440-44894947f8c6","Type":"ContainerDied","Data":"0bb81733d9e3c946826d726cb92877482626616f648c0920e3db44e84022cf7b"} Sep 11 18:20:45.490352 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:45.490157 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgs7td" event={"ID":"63719be4-37eb-4560-a440-44894947f8c6","Type":"ContainerStarted","Data":"e01506f1811543fe5cce3540259249c7c2d965ea4b38cbe418419f2b3c2e37f2"} Sep 11 18:20:48.507507 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:48.507465 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-4tfgc" event={"ID":"85438467-07eb-41d8-94c9-8dd0fd0f2975","Type":"ContainerStarted","Data":"2b6c055a0a83995c6b68f788c89ea8977c8e270cb2df116e120e255b73461229"} Sep 11 18:20:48.507980 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:48.507612 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-4tfgc" Sep 11 18:20:48.509312 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:48.509282 2561 generic.go:356] "Generic (PLEG): container finished" podID="63719be4-37eb-4560-a440-44894947f8c6" containerID="6ed2eb8284403b070c401ac9fa43fb0681856e691baad5ca5be991d4609c1b36" exitCode=0 Sep 11 18:20:48.509464 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:48.509323 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgs7td" event={"ID":"63719be4-37eb-4560-a440-44894947f8c6","Type":"ContainerDied","Data":"6ed2eb8284403b070c401ac9fa43fb0681856e691baad5ca5be991d4609c1b36"} Sep 11 18:20:48.529047 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:48.529000 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-4tfgc" podStartSLOduration=1.860897359 podStartE2EDuration="5.528984233s" totalImagesPullingTime="3.668086874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:20:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:20:48.527545472 +0000 UTC m=+330.024563227" watchObservedRunningTime="2026-09-11 18:20:48.528984233 +0000 UTC m=+330.026001987" Sep 11 18:20:49.515557 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:49.515525 2561 generic.go:356] "Generic (PLEG): container finished" podID="63719be4-37eb-4560-a440-44894947f8c6" containerID="8035d557e6d49b78b961956a6ba70680d733e29b406b719b486ea6b17e7a984c" exitCode=0 Sep 11 18:20:49.515941 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:49.515598 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgs7td" event={"ID":"63719be4-37eb-4560-a440-44894947f8c6","Type":"ContainerDied","Data":"8035d557e6d49b78b961956a6ba70680d733e29b406b719b486ea6b17e7a984c"} Sep 11 18:20:50.648243 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:50.648218 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgs7td" Sep 11 18:20:50.817883 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:50.817787 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wqb45\" (UniqueName: \"kubernetes.io/projected/63719be4-37eb-4560-a440-44894947f8c6-kube-api-access-wqb45\") pod \"63719be4-37eb-4560-a440-44894947f8c6\" (UID: \"63719be4-37eb-4560-a440-44894947f8c6\") " Sep 11 18:20:50.817883 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:50.817866 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/63719be4-37eb-4560-a440-44894947f8c6-util\") pod \"63719be4-37eb-4560-a440-44894947f8c6\" (UID: \"63719be4-37eb-4560-a440-44894947f8c6\") " Sep 11 18:20:50.818100 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:50.817896 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/63719be4-37eb-4560-a440-44894947f8c6-bundle\") pod \"63719be4-37eb-4560-a440-44894947f8c6\" (UID: \"63719be4-37eb-4560-a440-44894947f8c6\") " Sep 11 18:20:50.818315 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:50.818279 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63719be4-37eb-4560-a440-44894947f8c6-bundle" (OuterVolumeSpecName: "bundle") pod "63719be4-37eb-4560-a440-44894947f8c6" (UID: "63719be4-37eb-4560-a440-44894947f8c6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:20:50.819915 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:50.819883 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63719be4-37eb-4560-a440-44894947f8c6-kube-api-access-wqb45" (OuterVolumeSpecName: "kube-api-access-wqb45") pod "63719be4-37eb-4560-a440-44894947f8c6" (UID: "63719be4-37eb-4560-a440-44894947f8c6"). InnerVolumeSpecName "kube-api-access-wqb45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:20:50.822400 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:50.822376 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63719be4-37eb-4560-a440-44894947f8c6-util" (OuterVolumeSpecName: "util") pod "63719be4-37eb-4560-a440-44894947f8c6" (UID: "63719be4-37eb-4560-a440-44894947f8c6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:20:50.918679 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:50.918645 2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/63719be4-37eb-4560-a440-44894947f8c6-util\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:20:50.918841 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:50.918694 2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/63719be4-37eb-4560-a440-44894947f8c6-bundle\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:20:50.918841 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:50.918703 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqb45\" (UniqueName: \"kubernetes.io/projected/63719be4-37eb-4560-a440-44894947f8c6-kube-api-access-wqb45\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:20:51.526500 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:51.526472 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgs7td" Sep 11 18:20:51.526700 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:51.526470 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgs7td" event={"ID":"63719be4-37eb-4560-a440-44894947f8c6","Type":"ContainerDied","Data":"e01506f1811543fe5cce3540259249c7c2d965ea4b38cbe418419f2b3c2e37f2"} Sep 11 18:20:51.526700 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:51.526591 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e01506f1811543fe5cce3540259249c7c2d965ea4b38cbe418419f2b3c2e37f2" Sep 11 18:20:54.517602 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:20:54.517571 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-4tfgc" Sep 11 18:21:01.249603 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:01.249564 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-g4bsz"] Sep 11 18:21:01.250079 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:01.249883 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63719be4-37eb-4560-a440-44894947f8c6" containerName="util" Sep 11 18:21:01.250079 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:01.249896 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="63719be4-37eb-4560-a440-44894947f8c6" containerName="util" Sep 11 18:21:01.250079 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:01.249905 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63719be4-37eb-4560-a440-44894947f8c6" containerName="pull" Sep 11 18:21:01.250079 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:01.249911 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="63719be4-37eb-4560-a440-44894947f8c6" containerName="pull" Sep 11 18:21:01.250079 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:01.249929 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63719be4-37eb-4560-a440-44894947f8c6" containerName="extract" Sep 11 18:21:01.250079 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:01.249935 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="63719be4-37eb-4560-a440-44894947f8c6" containerName="extract" Sep 11 18:21:01.250079 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:01.249979 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="63719be4-37eb-4560-a440-44894947f8c6" containerName="extract" Sep 11 18:21:01.305728 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:01.305685 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-g4bsz"] Sep 11 18:21:01.305891 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:01.305790 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-g4bsz" Sep 11 18:21:01.308001 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:01.307973 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 11 18:21:01.308140 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:01.308015 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-2zxfc\"" Sep 11 18:21:01.308910 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:01.308894 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 11 18:21:01.405938 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:01.405899 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/850b2548-8f04-42ef-ab35-bc6a01bc0053-tmp\") pod \"openshift-lws-operator-fd8ccff4c-g4bsz\" (UID: \"850b2548-8f04-42ef-ab35-bc6a01bc0053\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-g4bsz" Sep 11 18:21:01.405938 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:01.405941 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx2h7\" (UniqueName: \"kubernetes.io/projected/850b2548-8f04-42ef-ab35-bc6a01bc0053-kube-api-access-rx2h7\") pod \"openshift-lws-operator-fd8ccff4c-g4bsz\" (UID: \"850b2548-8f04-42ef-ab35-bc6a01bc0053\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-g4bsz" Sep 11 18:21:01.506427 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:01.506329 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/850b2548-8f04-42ef-ab35-bc6a01bc0053-tmp\") pod \"openshift-lws-operator-fd8ccff4c-g4bsz\" (UID: \"850b2548-8f04-42ef-ab35-bc6a01bc0053\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-g4bsz" Sep 11 18:21:01.506427 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:01.506401 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx2h7\" (UniqueName: \"kubernetes.io/projected/850b2548-8f04-42ef-ab35-bc6a01bc0053-kube-api-access-rx2h7\") pod \"openshift-lws-operator-fd8ccff4c-g4bsz\" (UID: \"850b2548-8f04-42ef-ab35-bc6a01bc0053\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-g4bsz" Sep 11 18:21:01.506754 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:01.506733 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/850b2548-8f04-42ef-ab35-bc6a01bc0053-tmp\") pod \"openshift-lws-operator-fd8ccff4c-g4bsz\" (UID: \"850b2548-8f04-42ef-ab35-bc6a01bc0053\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-g4bsz" Sep 11 18:21:01.514846 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:01.514815 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx2h7\" (UniqueName: \"kubernetes.io/projected/850b2548-8f04-42ef-ab35-bc6a01bc0053-kube-api-access-rx2h7\") pod \"openshift-lws-operator-fd8ccff4c-g4bsz\" (UID: \"850b2548-8f04-42ef-ab35-bc6a01bc0053\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-g4bsz" Sep 11 18:21:01.615488 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:01.615449 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-g4bsz" Sep 11 18:21:01.909807 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:01.909779 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-g4bsz"] Sep 11 18:21:01.912263 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:21:01.912239 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod850b2548_8f04_42ef_ab35_bc6a01bc0053.slice/crio-4853ff443ab144220c4ca5eed72477ca714990229c600b7f125f4eb323e415b8 WatchSource:0}: Error finding container 4853ff443ab144220c4ca5eed72477ca714990229c600b7f125f4eb323e415b8: Status 404 returned error can't find the container with id 4853ff443ab144220c4ca5eed72477ca714990229c600b7f125f4eb323e415b8 Sep 11 18:21:02.577521 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:02.577485 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-g4bsz" event={"ID":"850b2548-8f04-42ef-ab35-bc6a01bc0053","Type":"ContainerStarted","Data":"4853ff443ab144220c4ca5eed72477ca714990229c600b7f125f4eb323e415b8"} Sep 11 18:21:04.589327 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:04.589286 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-g4bsz" event={"ID":"850b2548-8f04-42ef-ab35-bc6a01bc0053","Type":"ContainerStarted","Data":"e29db365b74bcbdec66610dd723a300aa4623c2c5c26d3f1f9b3feda821a2288"} Sep 11 18:21:04.607412 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:04.607352 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-g4bsz" podStartSLOduration=1.49468185 podStartE2EDuration="3.607335841s" totalImagesPullingTime="2.112653991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:21:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:21:04.6051613 +0000 UTC m=+346.102179054" watchObservedRunningTime="2026-09-11 18:21:04.607335841 +0000 UTC m=+346.104353594" Sep 11 18:21:09.027297 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:09.027262 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhjkz8"] Sep 11 18:21:09.029376 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:09.029358 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhjkz8" Sep 11 18:21:09.031972 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:09.031945 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-gfgp7\"" Sep 11 18:21:09.032121 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:09.031951 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 11 18:21:09.032639 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:09.032622 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 11 18:21:09.041296 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:09.041272 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhjkz8"] Sep 11 18:21:09.076602 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:09.076555 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a19a0b01-af7e-4904-bb85-b37f13d40ee2-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhjkz8\" (UID: \"a19a0b01-af7e-4904-bb85-b37f13d40ee2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhjkz8" Sep 11 18:21:09.076779 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:09.076619 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a19a0b01-af7e-4904-bb85-b37f13d40ee2-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhjkz8\" (UID: \"a19a0b01-af7e-4904-bb85-b37f13d40ee2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhjkz8" Sep 11 18:21:09.076779 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:09.076644 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79l22\" (UniqueName: \"kubernetes.io/projected/a19a0b01-af7e-4904-bb85-b37f13d40ee2-kube-api-access-79l22\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhjkz8\" (UID: \"a19a0b01-af7e-4904-bb85-b37f13d40ee2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhjkz8" Sep 11 18:21:09.177117 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:09.177071 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a19a0b01-af7e-4904-bb85-b37f13d40ee2-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhjkz8\" (UID: \"a19a0b01-af7e-4904-bb85-b37f13d40ee2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhjkz8" Sep 11 18:21:09.177309 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:09.177133 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a19a0b01-af7e-4904-bb85-b37f13d40ee2-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhjkz8\" (UID: \"a19a0b01-af7e-4904-bb85-b37f13d40ee2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhjkz8" Sep 11 18:21:09.177309 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:09.177160 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79l22\" (UniqueName: \"kubernetes.io/projected/a19a0b01-af7e-4904-bb85-b37f13d40ee2-kube-api-access-79l22\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhjkz8\" (UID: \"a19a0b01-af7e-4904-bb85-b37f13d40ee2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhjkz8" Sep 11 18:21:09.177533 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:09.177513 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a19a0b01-af7e-4904-bb85-b37f13d40ee2-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhjkz8\" (UID: \"a19a0b01-af7e-4904-bb85-b37f13d40ee2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhjkz8" Sep 11 18:21:09.177619 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:09.177568 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a19a0b01-af7e-4904-bb85-b37f13d40ee2-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhjkz8\" (UID: \"a19a0b01-af7e-4904-bb85-b37f13d40ee2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhjkz8" Sep 11 18:21:09.186400 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:09.186372 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79l22\" (UniqueName: \"kubernetes.io/projected/a19a0b01-af7e-4904-bb85-b37f13d40ee2-kube-api-access-79l22\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhjkz8\" (UID: \"a19a0b01-af7e-4904-bb85-b37f13d40ee2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhjkz8" Sep 11 18:21:09.338404 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:09.338314 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhjkz8" Sep 11 18:21:09.613163 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:09.613141 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhjkz8"] Sep 11 18:21:09.615336 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:21:09.615307 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda19a0b01_af7e_4904_bb85_b37f13d40ee2.slice/crio-454f29358f567a56cfce7d4c56db4df57d1675f12e9132ef3d27873501ed22a3 WatchSource:0}: Error finding container 454f29358f567a56cfce7d4c56db4df57d1675f12e9132ef3d27873501ed22a3: Status 404 returned error can't find the container with id 454f29358f567a56cfce7d4c56db4df57d1675f12e9132ef3d27873501ed22a3 Sep 11 18:21:10.616813 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:10.616721 2561 generic.go:356] "Generic (PLEG): container finished" podID="a19a0b01-af7e-4904-bb85-b37f13d40ee2" containerID="dac2d64226b70bee923ed93cc89d67f8f7cd20c1ece0771f55493113b56738c3" exitCode=0 Sep 11 18:21:10.617190 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:10.616805 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhjkz8" event={"ID":"a19a0b01-af7e-4904-bb85-b37f13d40ee2","Type":"ContainerDied","Data":"dac2d64226b70bee923ed93cc89d67f8f7cd20c1ece0771f55493113b56738c3"} Sep 11 18:21:10.617190 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:10.616885 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhjkz8" event={"ID":"a19a0b01-af7e-4904-bb85-b37f13d40ee2","Type":"ContainerStarted","Data":"454f29358f567a56cfce7d4c56db4df57d1675f12e9132ef3d27873501ed22a3"} Sep 11 18:21:11.625717 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:11.625623 2561 generic.go:356] "Generic (PLEG): container finished" podID="a19a0b01-af7e-4904-bb85-b37f13d40ee2" containerID="48ac0bb93f6d47b1d1098e919807a6e3d7fb4b0ec9ed768fdcbc0b46633dfe3c" exitCode=0 Sep 11 18:21:11.626074 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:11.625711 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhjkz8" event={"ID":"a19a0b01-af7e-4904-bb85-b37f13d40ee2","Type":"ContainerDied","Data":"48ac0bb93f6d47b1d1098e919807a6e3d7fb4b0ec9ed768fdcbc0b46633dfe3c"} Sep 11 18:21:12.632169 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:12.632134 2561 generic.go:356] "Generic (PLEG): container finished" podID="a19a0b01-af7e-4904-bb85-b37f13d40ee2" containerID="56d9d4de5524e33b2be205f32473ec296403b55a32c4d47ca230962a8eb1643b" exitCode=0 Sep 11 18:21:12.632543 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:12.632217 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhjkz8" event={"ID":"a19a0b01-af7e-4904-bb85-b37f13d40ee2","Type":"ContainerDied","Data":"56d9d4de5524e33b2be205f32473ec296403b55a32c4d47ca230962a8eb1643b"} Sep 11 18:21:13.759004 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:13.758969 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhjkz8" Sep 11 18:21:13.811840 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:13.811802 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a19a0b01-af7e-4904-bb85-b37f13d40ee2-bundle\") pod \"a19a0b01-af7e-4904-bb85-b37f13d40ee2\" (UID: \"a19a0b01-af7e-4904-bb85-b37f13d40ee2\") " Sep 11 18:21:13.812030 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:13.811856 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-79l22\" (UniqueName: \"kubernetes.io/projected/a19a0b01-af7e-4904-bb85-b37f13d40ee2-kube-api-access-79l22\") pod \"a19a0b01-af7e-4904-bb85-b37f13d40ee2\" (UID: \"a19a0b01-af7e-4904-bb85-b37f13d40ee2\") " Sep 11 18:21:13.812030 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:13.811886 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a19a0b01-af7e-4904-bb85-b37f13d40ee2-util\") pod \"a19a0b01-af7e-4904-bb85-b37f13d40ee2\" (UID: \"a19a0b01-af7e-4904-bb85-b37f13d40ee2\") " Sep 11 18:21:13.812614 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:13.812588 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a19a0b01-af7e-4904-bb85-b37f13d40ee2-bundle" (OuterVolumeSpecName: "bundle") pod "a19a0b01-af7e-4904-bb85-b37f13d40ee2" (UID: "a19a0b01-af7e-4904-bb85-b37f13d40ee2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:21:13.814137 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:13.814112 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a19a0b01-af7e-4904-bb85-b37f13d40ee2-kube-api-access-79l22" (OuterVolumeSpecName: "kube-api-access-79l22") pod "a19a0b01-af7e-4904-bb85-b37f13d40ee2" (UID: "a19a0b01-af7e-4904-bb85-b37f13d40ee2"). InnerVolumeSpecName "kube-api-access-79l22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:21:13.817509 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:13.817486 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a19a0b01-af7e-4904-bb85-b37f13d40ee2-util" (OuterVolumeSpecName: "util") pod "a19a0b01-af7e-4904-bb85-b37f13d40ee2" (UID: "a19a0b01-af7e-4904-bb85-b37f13d40ee2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:21:13.912948 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:13.912912 2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a19a0b01-af7e-4904-bb85-b37f13d40ee2-bundle\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:21:13.912948 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:13.912945 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79l22\" (UniqueName: \"kubernetes.io/projected/a19a0b01-af7e-4904-bb85-b37f13d40ee2-kube-api-access-79l22\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:21:13.913210 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:13.912958 2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a19a0b01-af7e-4904-bb85-b37f13d40ee2-util\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:21:14.642117 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:14.642087 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhjkz8" Sep 11 18:21:14.642279 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:14.642113 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhjkz8" event={"ID":"a19a0b01-af7e-4904-bb85-b37f13d40ee2","Type":"ContainerDied","Data":"454f29358f567a56cfce7d4c56db4df57d1675f12e9132ef3d27873501ed22a3"} Sep 11 18:21:14.642279 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:14.642157 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="454f29358f567a56cfce7d4c56db4df57d1675f12e9132ef3d27873501ed22a3" Sep 11 18:21:18.474545 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:18.474493 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dpxgp"] Sep 11 18:21:18.475007 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:18.474919 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a19a0b01-af7e-4904-bb85-b37f13d40ee2" containerName="util" Sep 11 18:21:18.475007 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:18.474942 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a19a0b01-af7e-4904-bb85-b37f13d40ee2" containerName="util" Sep 11 18:21:18.475007 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:18.474956 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a19a0b01-af7e-4904-bb85-b37f13d40ee2" containerName="extract" Sep 11 18:21:18.475007 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:18.474963 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a19a0b01-af7e-4904-bb85-b37f13d40ee2" containerName="extract" Sep 11 18:21:18.475007 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:18.474981 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a19a0b01-af7e-4904-bb85-b37f13d40ee2" containerName="pull" Sep 11 18:21:18.475007 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:18.474989 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a19a0b01-af7e-4904-bb85-b37f13d40ee2" containerName="pull" Sep 11 18:21:18.475321 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:18.475092 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a19a0b01-af7e-4904-bb85-b37f13d40ee2" containerName="extract" Sep 11 18:21:18.478739 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:18.478717 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dpxgp" Sep 11 18:21:18.481101 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:18.481079 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-gfgp7\"" Sep 11 18:21:18.481101 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:18.481098 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 11 18:21:18.481280 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:18.481100 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 11 18:21:18.487197 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:18.487175 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dpxgp"] Sep 11 18:21:18.551350 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:18.551302 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9516cc0c-cd66-40e9-ab96-6ea7ac35c00e-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dpxgp\" (UID: \"9516cc0c-cd66-40e9-ab96-6ea7ac35c00e\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dpxgp" Sep 11 18:21:18.551524 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:18.551393 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9516cc0c-cd66-40e9-ab96-6ea7ac35c00e-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dpxgp\" (UID: \"9516cc0c-cd66-40e9-ab96-6ea7ac35c00e\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dpxgp" Sep 11 18:21:18.551524 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:18.551465 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl79x\" (UniqueName: \"kubernetes.io/projected/9516cc0c-cd66-40e9-ab96-6ea7ac35c00e-kube-api-access-cl79x\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dpxgp\" (UID: \"9516cc0c-cd66-40e9-ab96-6ea7ac35c00e\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dpxgp" Sep 11 18:21:18.652336 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:18.652299 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9516cc0c-cd66-40e9-ab96-6ea7ac35c00e-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dpxgp\" (UID: \"9516cc0c-cd66-40e9-ab96-6ea7ac35c00e\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dpxgp" Sep 11 18:21:18.652560 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:18.652361 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9516cc0c-cd66-40e9-ab96-6ea7ac35c00e-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dpxgp\" (UID: \"9516cc0c-cd66-40e9-ab96-6ea7ac35c00e\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dpxgp" Sep 11 18:21:18.652560 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:18.652477 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl79x\" (UniqueName: \"kubernetes.io/projected/9516cc0c-cd66-40e9-ab96-6ea7ac35c00e-kube-api-access-cl79x\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dpxgp\" (UID: \"9516cc0c-cd66-40e9-ab96-6ea7ac35c00e\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dpxgp" Sep 11 18:21:18.652808 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:18.652778 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9516cc0c-cd66-40e9-ab96-6ea7ac35c00e-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dpxgp\" (UID: \"9516cc0c-cd66-40e9-ab96-6ea7ac35c00e\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dpxgp" Sep 11 18:21:18.652934 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:18.652841 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9516cc0c-cd66-40e9-ab96-6ea7ac35c00e-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dpxgp\" (UID: \"9516cc0c-cd66-40e9-ab96-6ea7ac35c00e\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dpxgp" Sep 11 18:21:18.664216 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:18.664181 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl79x\" (UniqueName: \"kubernetes.io/projected/9516cc0c-cd66-40e9-ab96-6ea7ac35c00e-kube-api-access-cl79x\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dpxgp\" (UID: \"9516cc0c-cd66-40e9-ab96-6ea7ac35c00e\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dpxgp" Sep 11 18:21:18.788191 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:18.788103 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dpxgp" Sep 11 18:21:19.061597 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:19.061570 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dpxgp"] Sep 11 18:21:19.064113 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:21:19.064083 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9516cc0c_cd66_40e9_ab96_6ea7ac35c00e.slice/crio-9229f582d3dce6a0ef107266f550de70a88fc0127416c188a2a33f169d0af07f WatchSource:0}: Error finding container 9229f582d3dce6a0ef107266f550de70a88fc0127416c188a2a33f169d0af07f: Status 404 returned error can't find the container with id 9229f582d3dce6a0ef107266f550de70a88fc0127416c188a2a33f169d0af07f Sep 11 18:21:19.664606 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:19.664572 2561 generic.go:356] "Generic (PLEG): container finished" podID="9516cc0c-cd66-40e9-ab96-6ea7ac35c00e" containerID="a26006cf30d1f5c08f6f718daab4965376081c2e88ac10341b98d5dc74b16584" exitCode=0 Sep 11 18:21:19.665216 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:19.664617 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dpxgp" event={"ID":"9516cc0c-cd66-40e9-ab96-6ea7ac35c00e","Type":"ContainerDied","Data":"a26006cf30d1f5c08f6f718daab4965376081c2e88ac10341b98d5dc74b16584"} Sep 11 18:21:19.665216 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:19.664663 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dpxgp" event={"ID":"9516cc0c-cd66-40e9-ab96-6ea7ac35c00e","Type":"ContainerStarted","Data":"9229f582d3dce6a0ef107266f550de70a88fc0127416c188a2a33f169d0af07f"} Sep 11 18:21:20.671086 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:20.671032 2561 generic.go:356] "Generic (PLEG): container finished" podID="9516cc0c-cd66-40e9-ab96-6ea7ac35c00e" containerID="d797901d61191f7cc512b0e74f5004dd166dc1d016dcb48c1df0484271ae23c1" exitCode=0 Sep 11 18:21:20.671503 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:20.671097 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dpxgp" event={"ID":"9516cc0c-cd66-40e9-ab96-6ea7ac35c00e","Type":"ContainerDied","Data":"d797901d61191f7cc512b0e74f5004dd166dc1d016dcb48c1df0484271ae23c1"} Sep 11 18:21:20.819619 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:20.819562 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-rg86p"] Sep 11 18:21:20.823308 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:20.823283 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-rg86p" Sep 11 18:21:20.826399 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:20.826373 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 11 18:21:20.826650 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:20.826632 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 11 18:21:20.826722 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:20.826652 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 11 18:21:20.826918 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:20.826901 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 11 18:21:20.827073 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:20.827041 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-8v878\"" Sep 11 18:21:20.842111 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:20.842077 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-rg86p"] Sep 11 18:21:20.872870 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:20.872833 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a5a6a746-6467-4629-b647-250c51a1e91f-manifests\") pod \"opendatahub-operator-controller-manager-76cff9b6b6-rg86p\" (UID: \"a5a6a746-6467-4629-b647-250c51a1e91f\") " pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-rg86p" Sep 11 18:21:20.873010 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:20.872889 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a5a6a746-6467-4629-b647-250c51a1e91f-webhook-cert\") pod \"opendatahub-operator-controller-manager-76cff9b6b6-rg86p\" (UID: \"a5a6a746-6467-4629-b647-250c51a1e91f\") " pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-rg86p" Sep 11 18:21:20.873010 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:20.872962 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a5a6a746-6467-4629-b647-250c51a1e91f-tmp\") pod \"opendatahub-operator-controller-manager-76cff9b6b6-rg86p\" (UID: \"a5a6a746-6467-4629-b647-250c51a1e91f\") " pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-rg86p" Sep 11 18:21:20.873179 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:20.873025 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a5a6a746-6467-4629-b647-250c51a1e91f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-76cff9b6b6-rg86p\" (UID: \"a5a6a746-6467-4629-b647-250c51a1e91f\") " pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-rg86p" Sep 11 18:21:20.873179 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:20.873080 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbfk2\" (UniqueName: \"kubernetes.io/projected/a5a6a746-6467-4629-b647-250c51a1e91f-kube-api-access-zbfk2\") pod \"opendatahub-operator-controller-manager-76cff9b6b6-rg86p\" (UID: \"a5a6a746-6467-4629-b647-250c51a1e91f\") " pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-rg86p" Sep 11 18:21:20.974193 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:20.974097 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a5a6a746-6467-4629-b647-250c51a1e91f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-76cff9b6b6-rg86p\" (UID: \"a5a6a746-6467-4629-b647-250c51a1e91f\") " pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-rg86p" Sep 11 18:21:20.974193 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:20.974144 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbfk2\" (UniqueName: \"kubernetes.io/projected/a5a6a746-6467-4629-b647-250c51a1e91f-kube-api-access-zbfk2\") pod \"opendatahub-operator-controller-manager-76cff9b6b6-rg86p\" (UID: \"a5a6a746-6467-4629-b647-250c51a1e91f\") " pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-rg86p" Sep 11 18:21:20.974413 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:20.974197 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a5a6a746-6467-4629-b647-250c51a1e91f-manifests\") pod \"opendatahub-operator-controller-manager-76cff9b6b6-rg86p\" (UID: \"a5a6a746-6467-4629-b647-250c51a1e91f\") " pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-rg86p" Sep 11 18:21:20.974518 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:20.974472 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a5a6a746-6467-4629-b647-250c51a1e91f-webhook-cert\") pod \"opendatahub-operator-controller-manager-76cff9b6b6-rg86p\" (UID: \"a5a6a746-6467-4629-b647-250c51a1e91f\") " pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-rg86p" Sep 11 18:21:20.974638 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:20.974566 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a5a6a746-6467-4629-b647-250c51a1e91f-tmp\") pod \"opendatahub-operator-controller-manager-76cff9b6b6-rg86p\" (UID: \"a5a6a746-6467-4629-b647-250c51a1e91f\") " pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-rg86p" Sep 11 18:21:20.974686 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:20.974649 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a5a6a746-6467-4629-b647-250c51a1e91f-manifests\") pod \"opendatahub-operator-controller-manager-76cff9b6b6-rg86p\" (UID: \"a5a6a746-6467-4629-b647-250c51a1e91f\") " pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-rg86p" Sep 11 18:21:20.976668 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:20.976642 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a5a6a746-6467-4629-b647-250c51a1e91f-tmp\") pod \"opendatahub-operator-controller-manager-76cff9b6b6-rg86p\" (UID: \"a5a6a746-6467-4629-b647-250c51a1e91f\") " pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-rg86p" Sep 11 18:21:20.976770 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:20.976674 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a5a6a746-6467-4629-b647-250c51a1e91f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-76cff9b6b6-rg86p\" (UID: \"a5a6a746-6467-4629-b647-250c51a1e91f\") " pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-rg86p" Sep 11 18:21:20.976850 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:20.976831 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a5a6a746-6467-4629-b647-250c51a1e91f-webhook-cert\") pod \"opendatahub-operator-controller-manager-76cff9b6b6-rg86p\" (UID: \"a5a6a746-6467-4629-b647-250c51a1e91f\") " pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-rg86p" Sep 11 18:21:20.986313 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:20.986286 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbfk2\" (UniqueName: \"kubernetes.io/projected/a5a6a746-6467-4629-b647-250c51a1e91f-kube-api-access-zbfk2\") pod \"opendatahub-operator-controller-manager-76cff9b6b6-rg86p\" (UID: \"a5a6a746-6467-4629-b647-250c51a1e91f\") " pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-rg86p" Sep 11 18:21:21.185328 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:21.185295 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-rg86p" Sep 11 18:21:21.463739 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:21.463711 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-rg86p"] Sep 11 18:21:21.465898 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:21:21.465865 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5a6a746_6467_4629_b647_250c51a1e91f.slice/crio-881d2487dfdcb1cdc24a59792619186d555f9cda30d0a8f8648f122e166b6f4c WatchSource:0}: Error finding container 881d2487dfdcb1cdc24a59792619186d555f9cda30d0a8f8648f122e166b6f4c: Status 404 returned error can't find the container with id 881d2487dfdcb1cdc24a59792619186d555f9cda30d0a8f8648f122e166b6f4c Sep 11 18:21:21.678011 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:21.677963 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-rg86p" event={"ID":"a5a6a746-6467-4629-b647-250c51a1e91f","Type":"ContainerStarted","Data":"881d2487dfdcb1cdc24a59792619186d555f9cda30d0a8f8648f122e166b6f4c"} Sep 11 18:21:21.680571 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:21.680544 2561 generic.go:356] "Generic (PLEG): container finished" podID="9516cc0c-cd66-40e9-ab96-6ea7ac35c00e" containerID="b32cb843e1245dbb57403db6c994aa40d2dfc9660b16a12f522b0adca3b09d16" exitCode=0 Sep 11 18:21:21.680693 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:21.680596 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dpxgp" event={"ID":"9516cc0c-cd66-40e9-ab96-6ea7ac35c00e","Type":"ContainerDied","Data":"b32cb843e1245dbb57403db6c994aa40d2dfc9660b16a12f522b0adca3b09d16"} Sep 11 18:21:22.841238 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:22.841209 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dpxgp" Sep 11 18:21:22.891373 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:22.891332 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9516cc0c-cd66-40e9-ab96-6ea7ac35c00e-bundle\") pod \"9516cc0c-cd66-40e9-ab96-6ea7ac35c00e\" (UID: \"9516cc0c-cd66-40e9-ab96-6ea7ac35c00e\") " Sep 11 18:21:22.891539 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:22.891458 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cl79x\" (UniqueName: \"kubernetes.io/projected/9516cc0c-cd66-40e9-ab96-6ea7ac35c00e-kube-api-access-cl79x\") pod \"9516cc0c-cd66-40e9-ab96-6ea7ac35c00e\" (UID: \"9516cc0c-cd66-40e9-ab96-6ea7ac35c00e\") " Sep 11 18:21:22.891539 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:22.891518 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9516cc0c-cd66-40e9-ab96-6ea7ac35c00e-util\") pod \"9516cc0c-cd66-40e9-ab96-6ea7ac35c00e\" (UID: \"9516cc0c-cd66-40e9-ab96-6ea7ac35c00e\") " Sep 11 18:21:22.892212 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:22.892154 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9516cc0c-cd66-40e9-ab96-6ea7ac35c00e-bundle" (OuterVolumeSpecName: "bundle") pod "9516cc0c-cd66-40e9-ab96-6ea7ac35c00e" (UID: "9516cc0c-cd66-40e9-ab96-6ea7ac35c00e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:21:22.893933 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:22.893903 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9516cc0c-cd66-40e9-ab96-6ea7ac35c00e-kube-api-access-cl79x" (OuterVolumeSpecName: "kube-api-access-cl79x") pod "9516cc0c-cd66-40e9-ab96-6ea7ac35c00e" (UID: "9516cc0c-cd66-40e9-ab96-6ea7ac35c00e"). InnerVolumeSpecName "kube-api-access-cl79x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:21:22.898736 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:22.898712 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9516cc0c-cd66-40e9-ab96-6ea7ac35c00e-util" (OuterVolumeSpecName: "util") pod "9516cc0c-cd66-40e9-ab96-6ea7ac35c00e" (UID: "9516cc0c-cd66-40e9-ab96-6ea7ac35c00e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:21:22.992280 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:22.992188 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cl79x\" (UniqueName: \"kubernetes.io/projected/9516cc0c-cd66-40e9-ab96-6ea7ac35c00e-kube-api-access-cl79x\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:21:22.992280 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:22.992233 2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9516cc0c-cd66-40e9-ab96-6ea7ac35c00e-util\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:21:22.992280 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:22.992246 2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9516cc0c-cd66-40e9-ab96-6ea7ac35c00e-bundle\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:21:23.696165 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:23.696121 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dpxgp" event={"ID":"9516cc0c-cd66-40e9-ab96-6ea7ac35c00e","Type":"ContainerDied","Data":"9229f582d3dce6a0ef107266f550de70a88fc0127416c188a2a33f169d0af07f"} Sep 11 18:21:23.696165 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:23.696161 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0dpxgp" Sep 11 18:21:23.696386 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:23.696183 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9229f582d3dce6a0ef107266f550de70a88fc0127416c188a2a33f169d0af07f" Sep 11 18:21:25.707787 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:25.707754 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-rg86p" event={"ID":"a5a6a746-6467-4629-b647-250c51a1e91f","Type":"ContainerStarted","Data":"5fafadeb7c3dedbfb85bfb5f6dc2b14fdaeabae1ea4b7e74ba9b32c548c199f1"} Sep 11 18:21:26.713699 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:26.713661 2561 generic.go:356] "Generic (PLEG): container finished" podID="a5a6a746-6467-4629-b647-250c51a1e91f" containerID="5fafadeb7c3dedbfb85bfb5f6dc2b14fdaeabae1ea4b7e74ba9b32c548c199f1" exitCode=0 Sep 11 18:21:26.714120 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:26.713747 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-rg86p" event={"ID":"a5a6a746-6467-4629-b647-250c51a1e91f","Type":"ContainerDied","Data":"5fafadeb7c3dedbfb85bfb5f6dc2b14fdaeabae1ea4b7e74ba9b32c548c199f1"} Sep 11 18:21:27.719988 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:27.719951 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-rg86p" event={"ID":"a5a6a746-6467-4629-b647-250c51a1e91f","Type":"ContainerStarted","Data":"4e7b79cb2d9efcedaa1360860274a5f2873bd9334b55b08f7d8c4f3837027035"} Sep 11 18:21:27.720417 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:27.720110 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-rg86p" Sep 11 18:21:27.745577 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:27.745475 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-rg86p" podStartSLOduration=3.11299803 podStartE2EDuration="7.745423972s" totalImagesPullingTime="3.607192433s" totalInitContainerRuntime="1.025233509s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:21:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:21:27.744006583 +0000 UTC m=+369.241024338" watchObservedRunningTime="2026-09-11 18:21:27.745423972 +0000 UTC m=+369.242441726" Sep 11 18:21:38.726844 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:38.726811 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-rg86p" Sep 11 18:21:50.865886 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:50.865850 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-67fd44999f-t577s"] Sep 11 18:21:50.866414 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:50.866310 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9516cc0c-cd66-40e9-ab96-6ea7ac35c00e" containerName="util" Sep 11 18:21:50.866414 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:50.866329 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9516cc0c-cd66-40e9-ab96-6ea7ac35c00e" containerName="util" Sep 11 18:21:50.866414 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:50.866343 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9516cc0c-cd66-40e9-ab96-6ea7ac35c00e" containerName="pull" Sep 11 18:21:50.866414 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:50.866351 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9516cc0c-cd66-40e9-ab96-6ea7ac35c00e" containerName="pull" Sep 11 18:21:50.866414 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:50.866372 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9516cc0c-cd66-40e9-ab96-6ea7ac35c00e" containerName="extract" Sep 11 18:21:50.866414 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:50.866380 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9516cc0c-cd66-40e9-ab96-6ea7ac35c00e" containerName="extract" Sep 11 18:21:50.866719 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:50.866462 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9516cc0c-cd66-40e9-ab96-6ea7ac35c00e" containerName="extract" Sep 11 18:21:50.870768 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:50.870744 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-67fd44999f-t577s" Sep 11 18:21:50.873236 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:50.873206 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 11 18:21:50.873943 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:50.873922 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-slcmn\"" Sep 11 18:21:50.874046 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:50.874004 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 11 18:21:50.874046 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:50.874011 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 11 18:21:50.874422 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:50.874011 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 11 18:21:50.884292 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:50.884253 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-67fd44999f-t577s"] Sep 11 18:21:50.940327 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:50.940293 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c271705-ae8e-4715-bb87-d176e6e0ed07-tls-certs\") pod \"kube-auth-proxy-67fd44999f-t577s\" (UID: \"7c271705-ae8e-4715-bb87-d176e6e0ed07\") " pod="openshift-ingress/kube-auth-proxy-67fd44999f-t577s" Sep 11 18:21:50.940327 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:50.940340 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-772lg\" (UniqueName: \"kubernetes.io/projected/7c271705-ae8e-4715-bb87-d176e6e0ed07-kube-api-access-772lg\") pod \"kube-auth-proxy-67fd44999f-t577s\" (UID: \"7c271705-ae8e-4715-bb87-d176e6e0ed07\") " pod="openshift-ingress/kube-auth-proxy-67fd44999f-t577s" Sep 11 18:21:50.940549 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:50.940381 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c271705-ae8e-4715-bb87-d176e6e0ed07-tmp\") pod \"kube-auth-proxy-67fd44999f-t577s\" (UID: \"7c271705-ae8e-4715-bb87-d176e6e0ed07\") " pod="openshift-ingress/kube-auth-proxy-67fd44999f-t577s" Sep 11 18:21:51.041186 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:51.041148 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c271705-ae8e-4715-bb87-d176e6e0ed07-tmp\") pod \"kube-auth-proxy-67fd44999f-t577s\" (UID: \"7c271705-ae8e-4715-bb87-d176e6e0ed07\") " pod="openshift-ingress/kube-auth-proxy-67fd44999f-t577s" Sep 11 18:21:51.041395 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:51.041213 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c271705-ae8e-4715-bb87-d176e6e0ed07-tls-certs\") pod \"kube-auth-proxy-67fd44999f-t577s\" (UID: \"7c271705-ae8e-4715-bb87-d176e6e0ed07\") " pod="openshift-ingress/kube-auth-proxy-67fd44999f-t577s" Sep 11 18:21:51.041395 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:51.041243 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-772lg\" (UniqueName: \"kubernetes.io/projected/7c271705-ae8e-4715-bb87-d176e6e0ed07-kube-api-access-772lg\") pod \"kube-auth-proxy-67fd44999f-t577s\" (UID: \"7c271705-ae8e-4715-bb87-d176e6e0ed07\") " pod="openshift-ingress/kube-auth-proxy-67fd44999f-t577s" Sep 11 18:21:51.043498 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:51.043457 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c271705-ae8e-4715-bb87-d176e6e0ed07-tmp\") pod \"kube-auth-proxy-67fd44999f-t577s\" (UID: \"7c271705-ae8e-4715-bb87-d176e6e0ed07\") " pod="openshift-ingress/kube-auth-proxy-67fd44999f-t577s" Sep 11 18:21:51.043691 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:51.043671 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c271705-ae8e-4715-bb87-d176e6e0ed07-tls-certs\") pod \"kube-auth-proxy-67fd44999f-t577s\" (UID: \"7c271705-ae8e-4715-bb87-d176e6e0ed07\") " pod="openshift-ingress/kube-auth-proxy-67fd44999f-t577s" Sep 11 18:21:51.049773 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:51.049745 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-772lg\" (UniqueName: \"kubernetes.io/projected/7c271705-ae8e-4715-bb87-d176e6e0ed07-kube-api-access-772lg\") pod \"kube-auth-proxy-67fd44999f-t577s\" (UID: \"7c271705-ae8e-4715-bb87-d176e6e0ed07\") " pod="openshift-ingress/kube-auth-proxy-67fd44999f-t577s" Sep 11 18:21:51.181335 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:51.181292 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-67fd44999f-t577s" Sep 11 18:21:51.236561 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:51.236383 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-mnwhs"] Sep 11 18:21:51.241577 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:51.241443 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-mnwhs" Sep 11 18:21:51.244791 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:51.244350 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-mpzmf\"" Sep 11 18:21:51.244791 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:51.244610 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Sep 11 18:21:51.259386 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:51.259346 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-mnwhs"] Sep 11 18:21:51.343611 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:51.343578 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bf80fc01-7343-4f52-a954-dcf935983c7c-cert\") pod \"odh-model-controller-7998c67dc4-mnwhs\" (UID: \"bf80fc01-7343-4f52-a954-dcf935983c7c\") " pod="opendatahub/odh-model-controller-7998c67dc4-mnwhs" Sep 11 18:21:51.343780 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:51.343628 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8mpx\" (UniqueName: \"kubernetes.io/projected/bf80fc01-7343-4f52-a954-dcf935983c7c-kube-api-access-x8mpx\") pod \"odh-model-controller-7998c67dc4-mnwhs\" (UID: \"bf80fc01-7343-4f52-a954-dcf935983c7c\") " pod="opendatahub/odh-model-controller-7998c67dc4-mnwhs" Sep 11 18:21:51.444958 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:51.444867 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bf80fc01-7343-4f52-a954-dcf935983c7c-cert\") pod \"odh-model-controller-7998c67dc4-mnwhs\" (UID: \"bf80fc01-7343-4f52-a954-dcf935983c7c\") " pod="opendatahub/odh-model-controller-7998c67dc4-mnwhs" Sep 11 18:21:51.444958 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:51.444920 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8mpx\" (UniqueName: \"kubernetes.io/projected/bf80fc01-7343-4f52-a954-dcf935983c7c-kube-api-access-x8mpx\") pod \"odh-model-controller-7998c67dc4-mnwhs\" (UID: \"bf80fc01-7343-4f52-a954-dcf935983c7c\") " pod="opendatahub/odh-model-controller-7998c67dc4-mnwhs" Sep 11 18:21:51.445207 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:21:51.445020 2561 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Sep 11 18:21:51.445207 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:21:51.445114 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf80fc01-7343-4f52-a954-dcf935983c7c-cert podName:bf80fc01-7343-4f52-a954-dcf935983c7c nodeName:}" failed. No retries permitted until 2026-09-11 18:21:51.945096403 +0000 UTC m=+393.442114157 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/bf80fc01-7343-4f52-a954-dcf935983c7c-cert") pod "odh-model-controller-7998c67dc4-mnwhs" (UID: "bf80fc01-7343-4f52-a954-dcf935983c7c") : secret "odh-model-controller-webhook-cert" not found Sep 11 18:21:51.451818 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:51.451791 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-67fd44999f-t577s"] Sep 11 18:21:51.454336 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:21:51.454300 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c271705_ae8e_4715_bb87_d176e6e0ed07.slice/crio-f42e2b82716a941a28ef83d4819a09e6886b20e0d45f4e700d57d00b0635617a WatchSource:0}: Error finding container f42e2b82716a941a28ef83d4819a09e6886b20e0d45f4e700d57d00b0635617a: Status 404 returned error can't find the container with id f42e2b82716a941a28ef83d4819a09e6886b20e0d45f4e700d57d00b0635617a Sep 11 18:21:51.459872 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:51.459848 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8mpx\" (UniqueName: \"kubernetes.io/projected/bf80fc01-7343-4f52-a954-dcf935983c7c-kube-api-access-x8mpx\") pod \"odh-model-controller-7998c67dc4-mnwhs\" (UID: \"bf80fc01-7343-4f52-a954-dcf935983c7c\") " pod="opendatahub/odh-model-controller-7998c67dc4-mnwhs" Sep 11 18:21:51.835762 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:51.835664 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-67fd44999f-t577s" event={"ID":"7c271705-ae8e-4715-bb87-d176e6e0ed07","Type":"ContainerStarted","Data":"f42e2b82716a941a28ef83d4819a09e6886b20e0d45f4e700d57d00b0635617a"} Sep 11 18:21:51.897876 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:51.897836 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-l6vbm"] Sep 11 18:21:51.901460 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:51.901434 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-l6vbm" Sep 11 18:21:51.903630 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:51.903604 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Sep 11 18:21:51.912078 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:51.911998 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-l6vbm"] Sep 11 18:21:51.950288 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:51.950253 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpr9g\" (UniqueName: \"kubernetes.io/projected/b9832d55-a490-4d4e-8b98-3b4f676f42e9-kube-api-access-dpr9g\") pod \"model-serving-api-54d89bd5fc-l6vbm\" (UID: \"b9832d55-a490-4d4e-8b98-3b4f676f42e9\") " pod="opendatahub/model-serving-api-54d89bd5fc-l6vbm" Sep 11 18:21:51.950462 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:51.950338 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9832d55-a490-4d4e-8b98-3b4f676f42e9-tls-certs\") pod \"model-serving-api-54d89bd5fc-l6vbm\" (UID: \"b9832d55-a490-4d4e-8b98-3b4f676f42e9\") " pod="opendatahub/model-serving-api-54d89bd5fc-l6vbm" Sep 11 18:21:51.950462 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:51.950394 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bf80fc01-7343-4f52-a954-dcf935983c7c-cert\") pod \"odh-model-controller-7998c67dc4-mnwhs\" (UID: \"bf80fc01-7343-4f52-a954-dcf935983c7c\") " pod="opendatahub/odh-model-controller-7998c67dc4-mnwhs" Sep 11 18:21:51.950540 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:21:51.950494 2561 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Sep 11 18:21:51.950575 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:21:51.950555 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf80fc01-7343-4f52-a954-dcf935983c7c-cert podName:bf80fc01-7343-4f52-a954-dcf935983c7c nodeName:}" failed. No retries permitted until 2026-09-11 18:21:52.950536044 +0000 UTC m=+394.447553781 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/bf80fc01-7343-4f52-a954-dcf935983c7c-cert") pod "odh-model-controller-7998c67dc4-mnwhs" (UID: "bf80fc01-7343-4f52-a954-dcf935983c7c") : secret "odh-model-controller-webhook-cert" not found Sep 11 18:21:52.051855 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:52.051794 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpr9g\" (UniqueName: \"kubernetes.io/projected/b9832d55-a490-4d4e-8b98-3b4f676f42e9-kube-api-access-dpr9g\") pod \"model-serving-api-54d89bd5fc-l6vbm\" (UID: \"b9832d55-a490-4d4e-8b98-3b4f676f42e9\") " pod="opendatahub/model-serving-api-54d89bd5fc-l6vbm" Sep 11 18:21:52.052449 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:52.051946 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9832d55-a490-4d4e-8b98-3b4f676f42e9-tls-certs\") pod \"model-serving-api-54d89bd5fc-l6vbm\" (UID: \"b9832d55-a490-4d4e-8b98-3b4f676f42e9\") " pod="opendatahub/model-serving-api-54d89bd5fc-l6vbm" Sep 11 18:21:52.052606 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:21:52.052266 2561 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Sep 11 18:21:52.052681 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:21:52.052615 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9832d55-a490-4d4e-8b98-3b4f676f42e9-tls-certs podName:b9832d55-a490-4d4e-8b98-3b4f676f42e9 nodeName:}" failed. No retries permitted until 2026-09-11 18:21:52.552590464 +0000 UTC m=+394.049608216 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/b9832d55-a490-4d4e-8b98-3b4f676f42e9-tls-certs") pod "model-serving-api-54d89bd5fc-l6vbm" (UID: "b9832d55-a490-4d4e-8b98-3b4f676f42e9") : secret "model-serving-api-tls" not found Sep 11 18:21:52.064124 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:52.064088 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpr9g\" (UniqueName: \"kubernetes.io/projected/b9832d55-a490-4d4e-8b98-3b4f676f42e9-kube-api-access-dpr9g\") pod \"model-serving-api-54d89bd5fc-l6vbm\" (UID: \"b9832d55-a490-4d4e-8b98-3b4f676f42e9\") " pod="opendatahub/model-serving-api-54d89bd5fc-l6vbm" Sep 11 18:21:52.088211 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:52.088127 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jn57k"] Sep 11 18:21:52.092013 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:52.091899 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jn57k" Sep 11 18:21:52.094683 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:52.094353 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 11 18:21:52.094865 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:52.094840 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 11 18:21:52.094984 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:52.094915 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 11 18:21:52.095107 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:52.095089 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-2gzbm\"" Sep 11 18:21:52.095217 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:52.094571 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 11 18:21:52.104751 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:52.104676 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jn57k"] Sep 11 18:21:52.153743 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:52.153687 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dbc5344a-cf13-436e-a728-91f290d0dca5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-jn57k\" (UID: \"dbc5344a-cf13-436e-a728-91f290d0dca5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jn57k" Sep 11 18:21:52.153963 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:52.153748 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dbc5344a-cf13-436e-a728-91f290d0dca5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-jn57k\" (UID: \"dbc5344a-cf13-436e-a728-91f290d0dca5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jn57k" Sep 11 18:21:52.153963 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:52.153800 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbc5344a-cf13-436e-a728-91f290d0dca5-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-jn57k\" (UID: \"dbc5344a-cf13-436e-a728-91f290d0dca5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jn57k" Sep 11 18:21:52.153963 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:52.153862 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckkzd\" (UniqueName: \"kubernetes.io/projected/dbc5344a-cf13-436e-a728-91f290d0dca5-kube-api-access-ckkzd\") pod \"istiod-openshift-gateway-7b7995d5cd-jn57k\" (UID: \"dbc5344a-cf13-436e-a728-91f290d0dca5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jn57k" Sep 11 18:21:52.153963 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:52.153891 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbc5344a-cf13-436e-a728-91f290d0dca5-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-jn57k\" (UID: \"dbc5344a-cf13-436e-a728-91f290d0dca5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jn57k" Sep 11 18:21:52.153963 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:52.153915 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dbc5344a-cf13-436e-a728-91f290d0dca5-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-jn57k\" (UID: \"dbc5344a-cf13-436e-a728-91f290d0dca5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jn57k" Sep 11 18:21:52.153963 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:52.153940 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dbc5344a-cf13-436e-a728-91f290d0dca5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-jn57k\" (UID: \"dbc5344a-cf13-436e-a728-91f290d0dca5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jn57k" Sep 11 18:21:52.256102 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:52.255609 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckkzd\" (UniqueName: \"kubernetes.io/projected/dbc5344a-cf13-436e-a728-91f290d0dca5-kube-api-access-ckkzd\") pod \"istiod-openshift-gateway-7b7995d5cd-jn57k\" (UID: \"dbc5344a-cf13-436e-a728-91f290d0dca5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jn57k" Sep 11 18:21:52.256102 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:52.255675 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbc5344a-cf13-436e-a728-91f290d0dca5-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-jn57k\" (UID: \"dbc5344a-cf13-436e-a728-91f290d0dca5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jn57k" Sep 11 18:21:52.256102 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:52.255703 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dbc5344a-cf13-436e-a728-91f290d0dca5-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-jn57k\" (UID: \"dbc5344a-cf13-436e-a728-91f290d0dca5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jn57k" Sep 11 18:21:52.256102 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:52.255728 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dbc5344a-cf13-436e-a728-91f290d0dca5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-jn57k\" (UID: \"dbc5344a-cf13-436e-a728-91f290d0dca5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jn57k" Sep 11 18:21:52.256102 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:52.255915 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dbc5344a-cf13-436e-a728-91f290d0dca5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-jn57k\" (UID: \"dbc5344a-cf13-436e-a728-91f290d0dca5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jn57k" Sep 11 18:21:52.256102 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:52.255942 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dbc5344a-cf13-436e-a728-91f290d0dca5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-jn57k\" (UID: \"dbc5344a-cf13-436e-a728-91f290d0dca5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jn57k" Sep 11 18:21:52.256102 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:52.255994 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbc5344a-cf13-436e-a728-91f290d0dca5-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-jn57k\" (UID: \"dbc5344a-cf13-436e-a728-91f290d0dca5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jn57k" Sep 11 18:21:52.257207 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:52.257175 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dbc5344a-cf13-436e-a728-91f290d0dca5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-jn57k\" (UID: \"dbc5344a-cf13-436e-a728-91f290d0dca5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jn57k" Sep 11 18:21:52.259278 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:52.259232 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dbc5344a-cf13-436e-a728-91f290d0dca5-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-jn57k\" (UID: \"dbc5344a-cf13-436e-a728-91f290d0dca5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jn57k" Sep 11 18:21:52.259278 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:52.259252 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dbc5344a-cf13-436e-a728-91f290d0dca5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-jn57k\" (UID: \"dbc5344a-cf13-436e-a728-91f290d0dca5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jn57k" Sep 11 18:21:52.259873 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:52.259832 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dbc5344a-cf13-436e-a728-91f290d0dca5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-jn57k\" (UID: \"dbc5344a-cf13-436e-a728-91f290d0dca5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jn57k" Sep 11 18:21:52.262142 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:52.262117 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbc5344a-cf13-436e-a728-91f290d0dca5-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-jn57k\" (UID: \"dbc5344a-cf13-436e-a728-91f290d0dca5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jn57k" Sep 11 18:21:52.264634 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:52.264602 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbc5344a-cf13-436e-a728-91f290d0dca5-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-jn57k\" (UID: \"dbc5344a-cf13-436e-a728-91f290d0dca5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jn57k" Sep 11 18:21:52.265832 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:52.265789 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckkzd\" (UniqueName: \"kubernetes.io/projected/dbc5344a-cf13-436e-a728-91f290d0dca5-kube-api-access-ckkzd\") pod \"istiod-openshift-gateway-7b7995d5cd-jn57k\" (UID: \"dbc5344a-cf13-436e-a728-91f290d0dca5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jn57k" Sep 11 18:21:52.408106 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:52.407999 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jn57k" Sep 11 18:21:52.560125 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:52.560080 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9832d55-a490-4d4e-8b98-3b4f676f42e9-tls-certs\") pod \"model-serving-api-54d89bd5fc-l6vbm\" (UID: \"b9832d55-a490-4d4e-8b98-3b4f676f42e9\") " pod="opendatahub/model-serving-api-54d89bd5fc-l6vbm" Sep 11 18:21:52.563405 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:52.563376 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9832d55-a490-4d4e-8b98-3b4f676f42e9-tls-certs\") pod \"model-serving-api-54d89bd5fc-l6vbm\" (UID: \"b9832d55-a490-4d4e-8b98-3b4f676f42e9\") " pod="opendatahub/model-serving-api-54d89bd5fc-l6vbm" Sep 11 18:21:52.704843 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:52.704810 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jn57k"] Sep 11 18:21:52.708107 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:21:52.708075 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbc5344a_cf13_436e_a728_91f290d0dca5.slice/crio-d8a8c852df020360f4286131c53994abc54631f6861d1cdb83bada30ae3413ee WatchSource:0}: Error finding container d8a8c852df020360f4286131c53994abc54631f6861d1cdb83bada30ae3413ee: Status 404 returned error can't find the container with id d8a8c852df020360f4286131c53994abc54631f6861d1cdb83bada30ae3413ee Sep 11 18:21:52.813772 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:52.813726 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-l6vbm" Sep 11 18:21:52.844380 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:52.844345 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jn57k" event={"ID":"dbc5344a-cf13-436e-a728-91f290d0dca5","Type":"ContainerStarted","Data":"d8a8c852df020360f4286131c53994abc54631f6861d1cdb83bada30ae3413ee"} Sep 11 18:21:52.964686 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:52.964592 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bf80fc01-7343-4f52-a954-dcf935983c7c-cert\") pod \"odh-model-controller-7998c67dc4-mnwhs\" (UID: \"bf80fc01-7343-4f52-a954-dcf935983c7c\") " pod="opendatahub/odh-model-controller-7998c67dc4-mnwhs" Sep 11 18:21:52.967553 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:52.967520 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bf80fc01-7343-4f52-a954-dcf935983c7c-cert\") pod \"odh-model-controller-7998c67dc4-mnwhs\" (UID: \"bf80fc01-7343-4f52-a954-dcf935983c7c\") " pod="opendatahub/odh-model-controller-7998c67dc4-mnwhs" Sep 11 18:21:53.059503 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:53.059466 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-mnwhs" Sep 11 18:21:54.417978 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:54.417939 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-mnwhs"] Sep 11 18:21:54.429362 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:54.429323 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-l6vbm"] Sep 11 18:21:54.615022 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:21:54.614978 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf80fc01_7343_4f52_a954_dcf935983c7c.slice/crio-eb0b0386ff4ceaf2ab0cd6003c13f2723cc95345bc754aff5a7bb58c8d0b5c8b WatchSource:0}: Error finding container eb0b0386ff4ceaf2ab0cd6003c13f2723cc95345bc754aff5a7bb58c8d0b5c8b: Status 404 returned error can't find the container with id eb0b0386ff4ceaf2ab0cd6003c13f2723cc95345bc754aff5a7bb58c8d0b5c8b Sep 11 18:21:54.615968 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:21:54.615932 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9832d55_a490_4d4e_8b98_3b4f676f42e9.slice/crio-5df1748188fa8660f2a04199ab436c6c42f68db7dd80192fb7c7ea223d1f41cf WatchSource:0}: Error finding container 5df1748188fa8660f2a04199ab436c6c42f68db7dd80192fb7c7ea223d1f41cf: Status 404 returned error can't find the container with id 5df1748188fa8660f2a04199ab436c6c42f68db7dd80192fb7c7ea223d1f41cf Sep 11 18:21:54.857012 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:54.856914 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-l6vbm" event={"ID":"b9832d55-a490-4d4e-8b98-3b4f676f42e9","Type":"ContainerStarted","Data":"5df1748188fa8660f2a04199ab436c6c42f68db7dd80192fb7c7ea223d1f41cf"} Sep 11 18:21:54.858535 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:54.858502 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-mnwhs" event={"ID":"bf80fc01-7343-4f52-a954-dcf935983c7c","Type":"ContainerStarted","Data":"eb0b0386ff4ceaf2ab0cd6003c13f2723cc95345bc754aff5a7bb58c8d0b5c8b"} Sep 11 18:21:54.860543 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:54.860498 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-67fd44999f-t577s" event={"ID":"7c271705-ae8e-4715-bb87-d176e6e0ed07","Type":"ContainerStarted","Data":"63445db75935f001c8a69bcd465b36829e83e9b4518e91c07c44cabdd3928b68"} Sep 11 18:21:54.877883 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:54.877830 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-67fd44999f-t577s" podStartSLOduration=2.125295725 podStartE2EDuration="4.877808637s" totalImagesPullingTime="2.752512912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:21:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:21:54.876778131 +0000 UTC m=+396.373795905" watchObservedRunningTime="2026-09-11 18:21:54.877808637 +0000 UTC m=+396.374826392" Sep 11 18:21:58.185308 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:58.185265 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:21:58.185568 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:58.185342 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:21:58.887701 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:58.887660 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jn57k" event={"ID":"dbc5344a-cf13-436e-a728-91f290d0dca5","Type":"ContainerStarted","Data":"dafd4f80837f92f853c79333d4c275acbd865e53bea34aaac3b117f7aad34241"} Sep 11 18:21:58.887894 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:58.887753 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jn57k" Sep 11 18:21:58.890260 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:58.890235 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-l6vbm" event={"ID":"b9832d55-a490-4d4e-8b98-3b4f676f42e9","Type":"ContainerStarted","Data":"eca3627dc8fddaaf3a8d195da559a05d28dfae022229c88d5a3a7e64e57d27b3"} Sep 11 18:21:58.890428 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:58.890409 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-l6vbm" Sep 11 18:21:58.892385 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:58.892362 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-mnwhs" event={"ID":"bf80fc01-7343-4f52-a954-dcf935983c7c","Type":"ContainerStarted","Data":"02ea18e1305879a8621ab4b70087c9ea72c6e998c1a974840bc8f0fd3926680c"} Sep 11 18:21:58.892479 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:58.892431 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-mnwhs" Sep 11 18:21:58.911377 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:58.911318 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jn57k" podStartSLOduration=1.437404988 podStartE2EDuration="6.911301398s" totalImagesPullingTime="5.47389641s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:21:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:21:58.908816316 +0000 UTC m=+400.405834071" watchObservedRunningTime="2026-09-11 18:21:58.911301398 +0000 UTC m=+400.408319153" Sep 11 18:21:58.930038 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:58.929975 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-l6vbm" podStartSLOduration=4.362602387 podStartE2EDuration="7.929954257s" totalImagesPullingTime="3.56735187s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:21:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:21:58.928905604 +0000 UTC m=+400.425923359" watchObservedRunningTime="2026-09-11 18:21:58.929954257 +0000 UTC m=+400.426972013" Sep 11 18:21:58.949435 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:58.949377 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-mnwhs" podStartSLOduration=4.341302699 podStartE2EDuration="7.949359771s" totalImagesPullingTime="3.608057072s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:21:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:21:58.948542303 +0000 UTC m=+400.445560081" watchObservedRunningTime="2026-09-11 18:21:58.949359771 +0000 UTC m=+400.446377516" Sep 11 18:21:59.700805 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:59.700766 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7"] Sep 11 18:21:59.704730 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:59.704700 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7" Sep 11 18:21:59.708666 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:59.708632 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-mpfgv\"" Sep 11 18:21:59.721402 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:59.721369 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7"] Sep 11 18:21:59.728451 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:59.728423 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61673abc-9544-4cf5-aad1-4714f9aacabc-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7\" (UID: \"61673abc-9544-4cf5-aad1-4714f9aacabc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7" Sep 11 18:21:59.728609 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:59.728463 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61673abc-9544-4cf5-aad1-4714f9aacabc-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7\" (UID: \"61673abc-9544-4cf5-aad1-4714f9aacabc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7" Sep 11 18:21:59.728609 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:59.728538 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61673abc-9544-4cf5-aad1-4714f9aacabc-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7\" (UID: \"61673abc-9544-4cf5-aad1-4714f9aacabc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7" Sep 11 18:21:59.728609 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:59.728595 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfd7l\" (UniqueName: \"kubernetes.io/projected/61673abc-9544-4cf5-aad1-4714f9aacabc-kube-api-access-kfd7l\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7\" (UID: \"61673abc-9544-4cf5-aad1-4714f9aacabc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7" Sep 11 18:21:59.728731 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:59.728659 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61673abc-9544-4cf5-aad1-4714f9aacabc-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7\" (UID: \"61673abc-9544-4cf5-aad1-4714f9aacabc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7" Sep 11 18:21:59.728731 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:59.728698 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61673abc-9544-4cf5-aad1-4714f9aacabc-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7\" (UID: \"61673abc-9544-4cf5-aad1-4714f9aacabc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7" Sep 11 18:21:59.728731 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:59.728718 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61673abc-9544-4cf5-aad1-4714f9aacabc-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7\" (UID: \"61673abc-9544-4cf5-aad1-4714f9aacabc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7" Sep 11 18:21:59.728845 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:59.728820 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61673abc-9544-4cf5-aad1-4714f9aacabc-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7\" (UID: \"61673abc-9544-4cf5-aad1-4714f9aacabc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7" Sep 11 18:21:59.728890 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:59.728876 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61673abc-9544-4cf5-aad1-4714f9aacabc-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7\" (UID: \"61673abc-9544-4cf5-aad1-4714f9aacabc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7" Sep 11 18:21:59.829840 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:59.829795 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61673abc-9544-4cf5-aad1-4714f9aacabc-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7\" (UID: \"61673abc-9544-4cf5-aad1-4714f9aacabc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7" Sep 11 18:21:59.830038 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:59.829857 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61673abc-9544-4cf5-aad1-4714f9aacabc-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7\" (UID: \"61673abc-9544-4cf5-aad1-4714f9aacabc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7" Sep 11 18:21:59.830038 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:59.829887 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61673abc-9544-4cf5-aad1-4714f9aacabc-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7\" (UID: \"61673abc-9544-4cf5-aad1-4714f9aacabc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7" Sep 11 18:21:59.830038 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:59.829912 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61673abc-9544-4cf5-aad1-4714f9aacabc-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7\" (UID: \"61673abc-9544-4cf5-aad1-4714f9aacabc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7" Sep 11 18:21:59.830038 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:59.829946 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61673abc-9544-4cf5-aad1-4714f9aacabc-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7\" (UID: \"61673abc-9544-4cf5-aad1-4714f9aacabc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7" Sep 11 18:21:59.830038 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:59.829986 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61673abc-9544-4cf5-aad1-4714f9aacabc-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7\" (UID: \"61673abc-9544-4cf5-aad1-4714f9aacabc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7" Sep 11 18:21:59.830038 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:59.830010 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61673abc-9544-4cf5-aad1-4714f9aacabc-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7\" (UID: \"61673abc-9544-4cf5-aad1-4714f9aacabc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7" Sep 11 18:21:59.830448 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:59.830041 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61673abc-9544-4cf5-aad1-4714f9aacabc-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7\" (UID: \"61673abc-9544-4cf5-aad1-4714f9aacabc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7" Sep 11 18:21:59.830448 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:59.830111 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfd7l\" (UniqueName: \"kubernetes.io/projected/61673abc-9544-4cf5-aad1-4714f9aacabc-kube-api-access-kfd7l\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7\" (UID: \"61673abc-9544-4cf5-aad1-4714f9aacabc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7" Sep 11 18:21:59.830448 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:59.830319 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61673abc-9544-4cf5-aad1-4714f9aacabc-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7\" (UID: \"61673abc-9544-4cf5-aad1-4714f9aacabc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7" Sep 11 18:21:59.830448 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:59.830344 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61673abc-9544-4cf5-aad1-4714f9aacabc-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7\" (UID: \"61673abc-9544-4cf5-aad1-4714f9aacabc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7" Sep 11 18:21:59.830657 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:59.830576 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61673abc-9544-4cf5-aad1-4714f9aacabc-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7\" (UID: \"61673abc-9544-4cf5-aad1-4714f9aacabc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7" Sep 11 18:21:59.830657 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:59.830643 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61673abc-9544-4cf5-aad1-4714f9aacabc-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7\" (UID: \"61673abc-9544-4cf5-aad1-4714f9aacabc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7" Sep 11 18:21:59.830766 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:59.830740 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61673abc-9544-4cf5-aad1-4714f9aacabc-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7\" (UID: \"61673abc-9544-4cf5-aad1-4714f9aacabc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7" Sep 11 18:21:59.832418 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:59.832395 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61673abc-9544-4cf5-aad1-4714f9aacabc-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7\" (UID: \"61673abc-9544-4cf5-aad1-4714f9aacabc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7" Sep 11 18:21:59.832593 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:59.832576 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61673abc-9544-4cf5-aad1-4714f9aacabc-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7\" (UID: \"61673abc-9544-4cf5-aad1-4714f9aacabc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7" Sep 11 18:21:59.839740 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:59.839714 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61673abc-9544-4cf5-aad1-4714f9aacabc-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7\" (UID: \"61673abc-9544-4cf5-aad1-4714f9aacabc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7" Sep 11 18:21:59.839903 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:59.839781 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfd7l\" (UniqueName: \"kubernetes.io/projected/61673abc-9544-4cf5-aad1-4714f9aacabc-kube-api-access-kfd7l\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7\" (UID: \"61673abc-9544-4cf5-aad1-4714f9aacabc\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7" Sep 11 18:21:59.899366 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:21:59.899335 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jn57k" Sep 11 18:22:00.015727 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:00.015634 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7" Sep 11 18:22:00.300753 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:00.300725 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7"] Sep 11 18:22:00.302451 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:22:00.302329 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61673abc_9544_4cf5_aad1_4714f9aacabc.slice/crio-ab3e3de6fcb88f8b688c270317395b0cc2c2e12495fcfef56aeec5d6a0e01f94 WatchSource:0}: Error finding container ab3e3de6fcb88f8b688c270317395b0cc2c2e12495fcfef56aeec5d6a0e01f94: Status 404 returned error can't find the container with id ab3e3de6fcb88f8b688c270317395b0cc2c2e12495fcfef56aeec5d6a0e01f94 Sep 11 18:22:00.903753 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:00.903711 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7" event={"ID":"61673abc-9544-4cf5-aad1-4714f9aacabc","Type":"ContainerStarted","Data":"ab3e3de6fcb88f8b688c270317395b0cc2c2e12495fcfef56aeec5d6a0e01f94"} Sep 11 18:22:01.274580 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:01.274490 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-rfphm"] Sep 11 18:22:01.280008 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:01.279981 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-rfphm" Sep 11 18:22:01.283164 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:01.283134 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-4mfd7\"" Sep 11 18:22:01.283411 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:01.283392 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Sep 11 18:22:01.290491 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:01.289923 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-rfphm"] Sep 11 18:22:01.293955 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:01.293929 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-tqgww"] Sep 11 18:22:01.297988 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:01.297836 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tqgww" Sep 11 18:22:01.300565 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:01.300540 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-v6zd5\"" Sep 11 18:22:01.300974 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:01.300658 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Sep 11 18:22:01.317930 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:01.317530 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-tqgww"] Sep 11 18:22:01.345748 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:01.345711 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pm7b\" (UniqueName: \"kubernetes.io/projected/052c1dcf-bad7-47b8-b5da-8a1ef2459f46-kube-api-access-7pm7b\") pod \"llmisvc-controller-manager-777c6bd7d5-tqgww\" (UID: \"052c1dcf-bad7-47b8-b5da-8a1ef2459f46\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tqgww" Sep 11 18:22:01.345914 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:01.345777 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stgwb\" (UniqueName: \"kubernetes.io/projected/1329cb52-610f-4b0f-8b8b-102288987992-kube-api-access-stgwb\") pod \"kserve-controller-manager-d6f4b65b8-rfphm\" (UID: \"1329cb52-610f-4b0f-8b8b-102288987992\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-rfphm" Sep 11 18:22:01.345914 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:01.345820 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/052c1dcf-bad7-47b8-b5da-8a1ef2459f46-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-tqgww\" (UID: \"052c1dcf-bad7-47b8-b5da-8a1ef2459f46\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tqgww" Sep 11 18:22:01.345914 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:01.345895 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1329cb52-610f-4b0f-8b8b-102288987992-cert\") pod \"kserve-controller-manager-d6f4b65b8-rfphm\" (UID: \"1329cb52-610f-4b0f-8b8b-102288987992\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-rfphm" Sep 11 18:22:01.447217 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:01.447180 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1329cb52-610f-4b0f-8b8b-102288987992-cert\") pod \"kserve-controller-manager-d6f4b65b8-rfphm\" (UID: \"1329cb52-610f-4b0f-8b8b-102288987992\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-rfphm" Sep 11 18:22:01.447407 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:01.447242 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pm7b\" (UniqueName: \"kubernetes.io/projected/052c1dcf-bad7-47b8-b5da-8a1ef2459f46-kube-api-access-7pm7b\") pod \"llmisvc-controller-manager-777c6bd7d5-tqgww\" (UID: \"052c1dcf-bad7-47b8-b5da-8a1ef2459f46\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tqgww" Sep 11 18:22:01.447407 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:01.447271 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stgwb\" (UniqueName: \"kubernetes.io/projected/1329cb52-610f-4b0f-8b8b-102288987992-kube-api-access-stgwb\") pod \"kserve-controller-manager-d6f4b65b8-rfphm\" (UID: \"1329cb52-610f-4b0f-8b8b-102288987992\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-rfphm" Sep 11 18:22:01.447407 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:01.447291 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/052c1dcf-bad7-47b8-b5da-8a1ef2459f46-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-tqgww\" (UID: \"052c1dcf-bad7-47b8-b5da-8a1ef2459f46\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tqgww" Sep 11 18:22:01.447407 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:22:01.447371 2561 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Sep 11 18:22:01.447623 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:22:01.447398 2561 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Sep 11 18:22:01.447623 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:22:01.447466 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1329cb52-610f-4b0f-8b8b-102288987992-cert podName:1329cb52-610f-4b0f-8b8b-102288987992 nodeName:}" failed. No retries permitted until 2026-09-11 18:22:01.947444118 +0000 UTC m=+403.444461867 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/1329cb52-610f-4b0f-8b8b-102288987992-cert") pod "kserve-controller-manager-d6f4b65b8-rfphm" (UID: "1329cb52-610f-4b0f-8b8b-102288987992") : secret "kserve-webhook-server-cert" not found Sep 11 18:22:01.447623 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:22:01.447486 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/052c1dcf-bad7-47b8-b5da-8a1ef2459f46-cert podName:052c1dcf-bad7-47b8-b5da-8a1ef2459f46 nodeName:}" failed. No retries permitted until 2026-09-11 18:22:01.947477324 +0000 UTC m=+403.444495058 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/052c1dcf-bad7-47b8-b5da-8a1ef2459f46-cert") pod "llmisvc-controller-manager-777c6bd7d5-tqgww" (UID: "052c1dcf-bad7-47b8-b5da-8a1ef2459f46") : secret "llmisvc-webhook-server-cert" not found Sep 11 18:22:01.457967 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:01.457929 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pm7b\" (UniqueName: \"kubernetes.io/projected/052c1dcf-bad7-47b8-b5da-8a1ef2459f46-kube-api-access-7pm7b\") pod \"llmisvc-controller-manager-777c6bd7d5-tqgww\" (UID: \"052c1dcf-bad7-47b8-b5da-8a1ef2459f46\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tqgww" Sep 11 18:22:01.467940 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:01.467894 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stgwb\" (UniqueName: \"kubernetes.io/projected/1329cb52-610f-4b0f-8b8b-102288987992-kube-api-access-stgwb\") pod \"kserve-controller-manager-d6f4b65b8-rfphm\" (UID: \"1329cb52-610f-4b0f-8b8b-102288987992\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-rfphm" Sep 11 18:22:01.953117 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:01.953082 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1329cb52-610f-4b0f-8b8b-102288987992-cert\") pod \"kserve-controller-manager-d6f4b65b8-rfphm\" (UID: \"1329cb52-610f-4b0f-8b8b-102288987992\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-rfphm" Sep 11 18:22:01.953506 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:01.953170 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/052c1dcf-bad7-47b8-b5da-8a1ef2459f46-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-tqgww\" (UID: \"052c1dcf-bad7-47b8-b5da-8a1ef2459f46\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tqgww" Sep 11 18:22:01.955711 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:01.955682 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/052c1dcf-bad7-47b8-b5da-8a1ef2459f46-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-tqgww\" (UID: \"052c1dcf-bad7-47b8-b5da-8a1ef2459f46\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tqgww" Sep 11 18:22:01.955837 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:01.955691 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1329cb52-610f-4b0f-8b8b-102288987992-cert\") pod \"kserve-controller-manager-d6f4b65b8-rfphm\" (UID: \"1329cb52-610f-4b0f-8b8b-102288987992\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-rfphm" Sep 11 18:22:02.195152 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:02.195104 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-rfphm" Sep 11 18:22:02.212209 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:02.212180 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tqgww" Sep 11 18:22:02.473840 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:02.473808 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-rfphm"] Sep 11 18:22:02.475131 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:22:02.475102 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1329cb52_610f_4b0f_8b8b_102288987992.slice/crio-2a30c28c1ffca2e3f10707dded8d36830dc6813be3e380e2d19f642a9cd9487e WatchSource:0}: Error finding container 2a30c28c1ffca2e3f10707dded8d36830dc6813be3e380e2d19f642a9cd9487e: Status 404 returned error can't find the container with id 2a30c28c1ffca2e3f10707dded8d36830dc6813be3e380e2d19f642a9cd9487e Sep 11 18:22:02.499431 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:02.499405 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-tqgww"] Sep 11 18:22:02.500903 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:22:02.500874 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod052c1dcf_bad7_47b8_b5da_8a1ef2459f46.slice/crio-44c246d1b514636eb5fd8686b59f72d65f776aa815618e85b3a7ae686c504822 WatchSource:0}: Error finding container 44c246d1b514636eb5fd8686b59f72d65f776aa815618e85b3a7ae686c504822: Status 404 returned error can't find the container with id 44c246d1b514636eb5fd8686b59f72d65f776aa815618e85b3a7ae686c504822 Sep 11 18:22:02.915432 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:02.915393 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tqgww" event={"ID":"052c1dcf-bad7-47b8-b5da-8a1ef2459f46","Type":"ContainerStarted","Data":"44c246d1b514636eb5fd8686b59f72d65f776aa815618e85b3a7ae686c504822"} Sep 11 18:22:02.916594 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:02.916569 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-rfphm" event={"ID":"1329cb52-610f-4b0f-8b8b-102288987992","Type":"ContainerStarted","Data":"2a30c28c1ffca2e3f10707dded8d36830dc6813be3e380e2d19f642a9cd9487e"} Sep 11 18:22:03.029508 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:03.029472 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jn57k"] Sep 11 18:22:03.029916 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:03.029674 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jn57k" podUID="dbc5344a-cf13-436e-a728-91f290d0dca5" containerName="discovery" containerID="cri-o://dafd4f80837f92f853c79333d4c275acbd865e53bea34aaac3b117f7aad34241" gracePeriod=30 Sep 11 18:22:03.282236 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:03.282190 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jn57k" Sep 11 18:22:03.363946 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:03.363903 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dbc5344a-cf13-436e-a728-91f290d0dca5-istio-csr-ca-configmap\") pod \"dbc5344a-cf13-436e-a728-91f290d0dca5\" (UID: \"dbc5344a-cf13-436e-a728-91f290d0dca5\") " Sep 11 18:22:03.364140 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:03.363957 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbc5344a-cf13-436e-a728-91f290d0dca5-istio-token\") pod \"dbc5344a-cf13-436e-a728-91f290d0dca5\" (UID: \"dbc5344a-cf13-436e-a728-91f290d0dca5\") " Sep 11 18:22:03.364140 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:03.364027 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dbc5344a-cf13-436e-a728-91f290d0dca5-istio-kubeconfig\") pod \"dbc5344a-cf13-436e-a728-91f290d0dca5\" (UID: \"dbc5344a-cf13-436e-a728-91f290d0dca5\") " Sep 11 18:22:03.364140 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:03.364095 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbc5344a-cf13-436e-a728-91f290d0dca5-local-certs\") pod \"dbc5344a-cf13-436e-a728-91f290d0dca5\" (UID: \"dbc5344a-cf13-436e-a728-91f290d0dca5\") " Sep 11 18:22:03.364308 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:03.364155 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dbc5344a-cf13-436e-a728-91f290d0dca5-cacerts\") pod \"dbc5344a-cf13-436e-a728-91f290d0dca5\" (UID: \"dbc5344a-cf13-436e-a728-91f290d0dca5\") " Sep 11 18:22:03.364308 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:03.364185 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ckkzd\" (UniqueName: \"kubernetes.io/projected/dbc5344a-cf13-436e-a728-91f290d0dca5-kube-api-access-ckkzd\") pod \"dbc5344a-cf13-436e-a728-91f290d0dca5\" (UID: \"dbc5344a-cf13-436e-a728-91f290d0dca5\") " Sep 11 18:22:03.364308 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:03.364209 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dbc5344a-cf13-436e-a728-91f290d0dca5-istio-csr-dns-cert\") pod \"dbc5344a-cf13-436e-a728-91f290d0dca5\" (UID: \"dbc5344a-cf13-436e-a728-91f290d0dca5\") " Sep 11 18:22:03.364461 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:03.364394 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbc5344a-cf13-436e-a728-91f290d0dca5-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "dbc5344a-cf13-436e-a728-91f290d0dca5" (UID: "dbc5344a-cf13-436e-a728-91f290d0dca5"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:22:03.364525 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:03.364471 2561 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dbc5344a-cf13-436e-a728-91f290d0dca5-istio-csr-ca-configmap\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:22:03.366934 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:03.366877 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbc5344a-cf13-436e-a728-91f290d0dca5-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "dbc5344a-cf13-436e-a728-91f290d0dca5" (UID: "dbc5344a-cf13-436e-a728-91f290d0dca5"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:22:03.367113 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:03.366984 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbc5344a-cf13-436e-a728-91f290d0dca5-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "dbc5344a-cf13-436e-a728-91f290d0dca5" (UID: "dbc5344a-cf13-436e-a728-91f290d0dca5"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:22:03.367198 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:03.367126 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbc5344a-cf13-436e-a728-91f290d0dca5-istio-token" (OuterVolumeSpecName: "istio-token") pod "dbc5344a-cf13-436e-a728-91f290d0dca5" (UID: "dbc5344a-cf13-436e-a728-91f290d0dca5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:22:03.367387 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:03.367356 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbc5344a-cf13-436e-a728-91f290d0dca5-kube-api-access-ckkzd" (OuterVolumeSpecName: "kube-api-access-ckkzd") pod "dbc5344a-cf13-436e-a728-91f290d0dca5" (UID: "dbc5344a-cf13-436e-a728-91f290d0dca5"). InnerVolumeSpecName "kube-api-access-ckkzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:22:03.367486 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:03.367456 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbc5344a-cf13-436e-a728-91f290d0dca5-cacerts" (OuterVolumeSpecName: "cacerts") pod "dbc5344a-cf13-436e-a728-91f290d0dca5" (UID: "dbc5344a-cf13-436e-a728-91f290d0dca5"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:22:03.367695 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:03.367674 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbc5344a-cf13-436e-a728-91f290d0dca5-local-certs" (OuterVolumeSpecName: "local-certs") pod "dbc5344a-cf13-436e-a728-91f290d0dca5" (UID: "dbc5344a-cf13-436e-a728-91f290d0dca5"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:22:03.466028 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:03.465990 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbc5344a-cf13-436e-a728-91f290d0dca5-istio-token\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:22:03.466028 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:03.466031 2561 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dbc5344a-cf13-436e-a728-91f290d0dca5-istio-kubeconfig\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:22:03.466244 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:03.466044 2561 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbc5344a-cf13-436e-a728-91f290d0dca5-local-certs\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:22:03.466244 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:03.466081 2561 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dbc5344a-cf13-436e-a728-91f290d0dca5-cacerts\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:22:03.466244 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:03.466096 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckkzd\" (UniqueName: \"kubernetes.io/projected/dbc5344a-cf13-436e-a728-91f290d0dca5-kube-api-access-ckkzd\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:22:03.466244 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:03.466110 2561 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dbc5344a-cf13-436e-a728-91f290d0dca5-istio-csr-dns-cert\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:22:03.925699 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:03.925657 2561 generic.go:356] "Generic (PLEG): container finished" podID="dbc5344a-cf13-436e-a728-91f290d0dca5" containerID="dafd4f80837f92f853c79333d4c275acbd865e53bea34aaac3b117f7aad34241" exitCode=0 Sep 11 18:22:03.925882 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:03.925714 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jn57k" event={"ID":"dbc5344a-cf13-436e-a728-91f290d0dca5","Type":"ContainerDied","Data":"dafd4f80837f92f853c79333d4c275acbd865e53bea34aaac3b117f7aad34241"} Sep 11 18:22:03.925882 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:03.925811 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jn57k" event={"ID":"dbc5344a-cf13-436e-a728-91f290d0dca5","Type":"ContainerDied","Data":"d8a8c852df020360f4286131c53994abc54631f6861d1cdb83bada30ae3413ee"} Sep 11 18:22:03.925882 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:03.925728 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jn57k" Sep 11 18:22:03.925882 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:03.925843 2561 scope.go:117] "RemoveContainer" containerID="dafd4f80837f92f853c79333d4c275acbd865e53bea34aaac3b117f7aad34241" Sep 11 18:22:03.956312 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:03.956273 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jn57k"] Sep 11 18:22:03.968507 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:03.968472 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jn57k"] Sep 11 18:22:03.995212 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:03.995191 2561 scope.go:117] "RemoveContainer" containerID="dafd4f80837f92f853c79333d4c275acbd865e53bea34aaac3b117f7aad34241" Sep 11 18:22:03.995580 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:22:03.995557 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dafd4f80837f92f853c79333d4c275acbd865e53bea34aaac3b117f7aad34241\": container with ID starting with dafd4f80837f92f853c79333d4c275acbd865e53bea34aaac3b117f7aad34241 not found: ID does not exist" containerID="dafd4f80837f92f853c79333d4c275acbd865e53bea34aaac3b117f7aad34241" Sep 11 18:22:03.995656 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:03.995594 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dafd4f80837f92f853c79333d4c275acbd865e53bea34aaac3b117f7aad34241"} err="failed to get container status \"dafd4f80837f92f853c79333d4c275acbd865e53bea34aaac3b117f7aad34241\": rpc error: code = NotFound desc = could not find container \"dafd4f80837f92f853c79333d4c275acbd865e53bea34aaac3b117f7aad34241\": container with ID starting with dafd4f80837f92f853c79333d4c275acbd865e53bea34aaac3b117f7aad34241 not found: ID does not exist" Sep 11 18:22:05.140401 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:05.140363 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbc5344a-cf13-436e-a728-91f290d0dca5" path="/var/lib/kubelet/pods/dbc5344a-cf13-436e-a728-91f290d0dca5/volumes" Sep 11 18:22:05.939966 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:05.939932 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tqgww" event={"ID":"052c1dcf-bad7-47b8-b5da-8a1ef2459f46","Type":"ContainerStarted","Data":"994d6b04c9a67882251ba599acd4e5b1ed55af6a0dd2ee5ea0527b6b5ecfb0bc"} Sep 11 18:22:05.940184 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:05.940032 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tqgww" Sep 11 18:22:05.941728 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:05.941699 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-rfphm" event={"ID":"1329cb52-610f-4b0f-8b8b-102288987992","Type":"ContainerStarted","Data":"68331bae7253f29b0a2def8265138de2702754632bdff26c37b2727bdd393480"} Sep 11 18:22:05.941846 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:05.941748 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-rfphm" Sep 11 18:22:05.957874 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:05.957823 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tqgww" podStartSLOduration=1.622269534 podStartE2EDuration="4.95780823s" totalImagesPullingTime="3.335538696s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:22:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:22:05.956970472 +0000 UTC m=+407.453988226" watchObservedRunningTime="2026-09-11 18:22:05.95780823 +0000 UTC m=+407.454825978" Sep 11 18:22:05.975635 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:05.974832 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-rfphm" podStartSLOduration=2.388619979 podStartE2EDuration="4.974812637s" totalImagesPullingTime="2.586192658s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:22:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:22:05.97449262 +0000 UTC m=+407.471510374" watchObservedRunningTime="2026-09-11 18:22:05.974812637 +0000 UTC m=+407.471830391" Sep 11 18:22:09.522627 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:09.522586 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:22:09.523004 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:09.522683 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:22:09.523004 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:09.522731 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:22:09.899729 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:09.899647 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-mnwhs" Sep 11 18:22:09.903972 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:09.903952 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-l6vbm" Sep 11 18:22:09.966522 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:09.966485 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7" event={"ID":"61673abc-9544-4cf5-aad1-4714f9aacabc","Type":"ContainerStarted","Data":"9d910c7a0323cd18b30afd658e789d247e4e5d5f9561420deb2ebae2ac481695"} Sep 11 18:22:09.987387 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:09.987323 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7" podStartSLOduration=1.770009406 podStartE2EDuration="10.987300376s" totalImagesPullingTime="9.21729097s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:21:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:22:09.984309648 +0000 UTC m=+411.481327412" watchObservedRunningTime="2026-09-11 18:22:09.987300376 +0000 UTC m=+411.484318131" Sep 11 18:22:10.016183 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:10.016144 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7" Sep 11 18:22:10.017728 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:10.017701 2561 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Sep 11 18:22:10.017796 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:10.017755 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7" podUID="61673abc-9544-4cf5-aad1-4714f9aacabc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Sep 11 18:22:11.020995 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:11.020961 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7" Sep 11 18:22:11.976495 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:11.976461 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7" Sep 11 18:22:11.977419 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:11.977396 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7" Sep 11 18:22:13.052795 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:13.052756 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-77bfcd69b7-n58zd"] Sep 11 18:22:13.053325 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:13.053302 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbc5344a-cf13-436e-a728-91f290d0dca5" containerName="discovery" Sep 11 18:22:13.053325 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:13.053330 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbc5344a-cf13-436e-a728-91f290d0dca5" containerName="discovery" Sep 11 18:22:13.053499 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:13.053423 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbc5344a-cf13-436e-a728-91f290d0dca5" containerName="discovery" Sep 11 18:22:13.055297 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:13.055275 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77bfcd69b7-n58zd" Sep 11 18:22:13.071942 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:13.071911 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77bfcd69b7-n58zd"] Sep 11 18:22:13.163746 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:13.163712 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmx52\" (UniqueName: \"kubernetes.io/projected/3467da4b-a896-486d-b322-dc59457d7578-kube-api-access-cmx52\") pod \"console-77bfcd69b7-n58zd\" (UID: \"3467da4b-a896-486d-b322-dc59457d7578\") " pod="openshift-console/console-77bfcd69b7-n58zd" Sep 11 18:22:13.163937 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:13.163756 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3467da4b-a896-486d-b322-dc59457d7578-console-config\") pod \"console-77bfcd69b7-n58zd\" (UID: \"3467da4b-a896-486d-b322-dc59457d7578\") " pod="openshift-console/console-77bfcd69b7-n58zd" Sep 11 18:22:13.163937 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:13.163774 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3467da4b-a896-486d-b322-dc59457d7578-oauth-serving-cert\") pod \"console-77bfcd69b7-n58zd\" (UID: \"3467da4b-a896-486d-b322-dc59457d7578\") " pod="openshift-console/console-77bfcd69b7-n58zd" Sep 11 18:22:13.163937 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:13.163798 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3467da4b-a896-486d-b322-dc59457d7578-console-oauth-config\") pod \"console-77bfcd69b7-n58zd\" (UID: \"3467da4b-a896-486d-b322-dc59457d7578\") " pod="openshift-console/console-77bfcd69b7-n58zd" Sep 11 18:22:13.163937 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:13.163909 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3467da4b-a896-486d-b322-dc59457d7578-service-ca\") pod \"console-77bfcd69b7-n58zd\" (UID: \"3467da4b-a896-486d-b322-dc59457d7578\") " pod="openshift-console/console-77bfcd69b7-n58zd" Sep 11 18:22:13.164127 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:13.163949 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3467da4b-a896-486d-b322-dc59457d7578-trusted-ca-bundle\") pod \"console-77bfcd69b7-n58zd\" (UID: \"3467da4b-a896-486d-b322-dc59457d7578\") " pod="openshift-console/console-77bfcd69b7-n58zd" Sep 11 18:22:13.164127 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:13.163969 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3467da4b-a896-486d-b322-dc59457d7578-console-serving-cert\") pod \"console-77bfcd69b7-n58zd\" (UID: \"3467da4b-a896-486d-b322-dc59457d7578\") " pod="openshift-console/console-77bfcd69b7-n58zd" Sep 11 18:22:13.265340 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:13.265300 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3467da4b-a896-486d-b322-dc59457d7578-service-ca\") pod \"console-77bfcd69b7-n58zd\" (UID: \"3467da4b-a896-486d-b322-dc59457d7578\") " pod="openshift-console/console-77bfcd69b7-n58zd" Sep 11 18:22:13.265340 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:13.265345 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3467da4b-a896-486d-b322-dc59457d7578-trusted-ca-bundle\") pod \"console-77bfcd69b7-n58zd\" (UID: \"3467da4b-a896-486d-b322-dc59457d7578\") " pod="openshift-console/console-77bfcd69b7-n58zd" Sep 11 18:22:13.265587 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:13.265371 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3467da4b-a896-486d-b322-dc59457d7578-console-serving-cert\") pod \"console-77bfcd69b7-n58zd\" (UID: \"3467da4b-a896-486d-b322-dc59457d7578\") " pod="openshift-console/console-77bfcd69b7-n58zd" Sep 11 18:22:13.265587 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:13.265417 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmx52\" (UniqueName: \"kubernetes.io/projected/3467da4b-a896-486d-b322-dc59457d7578-kube-api-access-cmx52\") pod \"console-77bfcd69b7-n58zd\" (UID: \"3467da4b-a896-486d-b322-dc59457d7578\") " pod="openshift-console/console-77bfcd69b7-n58zd" Sep 11 18:22:13.265587 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:13.265466 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3467da4b-a896-486d-b322-dc59457d7578-console-config\") pod \"console-77bfcd69b7-n58zd\" (UID: \"3467da4b-a896-486d-b322-dc59457d7578\") " pod="openshift-console/console-77bfcd69b7-n58zd" Sep 11 18:22:13.265587 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:13.265497 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3467da4b-a896-486d-b322-dc59457d7578-oauth-serving-cert\") pod \"console-77bfcd69b7-n58zd\" (UID: \"3467da4b-a896-486d-b322-dc59457d7578\") " pod="openshift-console/console-77bfcd69b7-n58zd" Sep 11 18:22:13.265587 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:13.265535 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3467da4b-a896-486d-b322-dc59457d7578-console-oauth-config\") pod \"console-77bfcd69b7-n58zd\" (UID: \"3467da4b-a896-486d-b322-dc59457d7578\") " pod="openshift-console/console-77bfcd69b7-n58zd" Sep 11 18:22:13.266192 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:13.266167 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3467da4b-a896-486d-b322-dc59457d7578-service-ca\") pod \"console-77bfcd69b7-n58zd\" (UID: \"3467da4b-a896-486d-b322-dc59457d7578\") " pod="openshift-console/console-77bfcd69b7-n58zd" Sep 11 18:22:13.266357 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:13.266334 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3467da4b-a896-486d-b322-dc59457d7578-console-config\") pod \"console-77bfcd69b7-n58zd\" (UID: \"3467da4b-a896-486d-b322-dc59457d7578\") " pod="openshift-console/console-77bfcd69b7-n58zd" Sep 11 18:22:13.266411 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:13.266358 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3467da4b-a896-486d-b322-dc59457d7578-trusted-ca-bundle\") pod \"console-77bfcd69b7-n58zd\" (UID: \"3467da4b-a896-486d-b322-dc59457d7578\") " pod="openshift-console/console-77bfcd69b7-n58zd" Sep 11 18:22:13.266411 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:13.266377 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3467da4b-a896-486d-b322-dc59457d7578-oauth-serving-cert\") pod \"console-77bfcd69b7-n58zd\" (UID: \"3467da4b-a896-486d-b322-dc59457d7578\") " pod="openshift-console/console-77bfcd69b7-n58zd" Sep 11 18:22:13.268039 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:13.268010 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3467da4b-a896-486d-b322-dc59457d7578-console-oauth-config\") pod \"console-77bfcd69b7-n58zd\" (UID: \"3467da4b-a896-486d-b322-dc59457d7578\") " pod="openshift-console/console-77bfcd69b7-n58zd" Sep 11 18:22:13.268039 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:13.268032 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3467da4b-a896-486d-b322-dc59457d7578-console-serving-cert\") pod \"console-77bfcd69b7-n58zd\" (UID: \"3467da4b-a896-486d-b322-dc59457d7578\") " pod="openshift-console/console-77bfcd69b7-n58zd" Sep 11 18:22:13.275005 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:13.274980 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmx52\" (UniqueName: \"kubernetes.io/projected/3467da4b-a896-486d-b322-dc59457d7578-kube-api-access-cmx52\") pod \"console-77bfcd69b7-n58zd\" (UID: \"3467da4b-a896-486d-b322-dc59457d7578\") " pod="openshift-console/console-77bfcd69b7-n58zd" Sep 11 18:22:13.372683 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:13.372577 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77bfcd69b7-n58zd" Sep 11 18:22:13.645684 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:13.645655 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77bfcd69b7-n58zd"] Sep 11 18:22:13.647329 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:22:13.647303 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3467da4b_a896_486d_b322_dc59457d7578.slice/crio-678cdabd816beb611e04a8df7d7636e658683b618b65c746035d5398de2f3ab3 WatchSource:0}: Error finding container 678cdabd816beb611e04a8df7d7636e658683b618b65c746035d5398de2f3ab3: Status 404 returned error can't find the container with id 678cdabd816beb611e04a8df7d7636e658683b618b65c746035d5398de2f3ab3 Sep 11 18:22:13.987171 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:13.987127 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77bfcd69b7-n58zd" event={"ID":"3467da4b-a896-486d-b322-dc59457d7578","Type":"ContainerStarted","Data":"b9ed82100132033b3c879291b38c696067499aee2bdcd25c6750c029cc8c7e6d"} Sep 11 18:22:13.987373 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:13.987185 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77bfcd69b7-n58zd" event={"ID":"3467da4b-a896-486d-b322-dc59457d7578","Type":"ContainerStarted","Data":"678cdabd816beb611e04a8df7d7636e658683b618b65c746035d5398de2f3ab3"} Sep 11 18:22:14.007819 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:14.007765 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-77bfcd69b7-n58zd" podStartSLOduration=1.007749012 podStartE2EDuration="1.007749012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:22:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:22:14.005325925 +0000 UTC m=+415.502343681" watchObservedRunningTime="2026-09-11 18:22:14.007749012 +0000 UTC m=+415.504766765" Sep 11 18:22:23.372946 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:23.372901 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-77bfcd69b7-n58zd" Sep 11 18:22:23.373420 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:23.372966 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-77bfcd69b7-n58zd" Sep 11 18:22:23.377822 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:23.377797 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-77bfcd69b7-n58zd" Sep 11 18:22:24.040675 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:24.039923 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-77bfcd69b7-n58zd" Sep 11 18:22:24.090354 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:24.090305 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bb5b74866-mtlnv"] Sep 11 18:22:36.948918 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:36.948882 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tqgww" Sep 11 18:22:36.951789 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:36.951771 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-rfphm" Sep 11 18:22:49.116369 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:49.116307 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-bb5b74866-mtlnv" podUID="e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5" containerName="console" containerID="cri-o://a512123646cfbeea74d4c342b5d92af674d744fa0b7e1b05af35e896ada70393" gracePeriod=15 Sep 11 18:22:49.361715 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:49.361683 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bb5b74866-mtlnv" Sep 11 18:22:49.463744 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:49.463706 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xftw\" (UniqueName: \"kubernetes.io/projected/e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5-kube-api-access-5xftw\") pod \"e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5\" (UID: \"e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5\") " Sep 11 18:22:49.463933 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:49.463768 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5-console-serving-cert\") pod \"e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5\" (UID: \"e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5\") " Sep 11 18:22:49.463933 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:49.463790 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5-console-oauth-config\") pod \"e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5\" (UID: \"e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5\") " Sep 11 18:22:49.463933 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:49.463813 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5-service-ca\") pod \"e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5\" (UID: \"e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5\") " Sep 11 18:22:49.463933 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:49.463857 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5-console-config\") pod \"e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5\" (UID: \"e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5\") " Sep 11 18:22:49.463933 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:49.463881 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5-trusted-ca-bundle\") pod \"e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5\" (UID: \"e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5\") " Sep 11 18:22:49.463933 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:49.463924 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5-oauth-serving-cert\") pod \"e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5\" (UID: \"e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5\") " Sep 11 18:22:49.464384 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:49.464346 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5-console-config" (OuterVolumeSpecName: "console-config") pod "e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5" (UID: "e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:22:49.464505 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:49.464377 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5" (UID: "e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:22:49.464505 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:49.464388 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5" (UID: "e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:22:49.464505 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:49.464348 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5-service-ca" (OuterVolumeSpecName: "service-ca") pod "e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5" (UID: "e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:22:49.466000 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:49.465971 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5" (UID: "e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:22:49.466376 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:49.466347 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5-kube-api-access-5xftw" (OuterVolumeSpecName: "kube-api-access-5xftw") pod "e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5" (UID: "e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5"). InnerVolumeSpecName "kube-api-access-5xftw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:22:49.466479 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:49.466460 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5" (UID: "e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:22:49.564980 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:49.564936 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xftw\" (UniqueName: \"kubernetes.io/projected/e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5-kube-api-access-5xftw\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:22:49.564980 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:49.564971 2561 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5-console-serving-cert\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:22:49.564980 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:49.564982 2561 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5-console-oauth-config\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:22:49.564980 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:49.564991 2561 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5-service-ca\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:22:49.565278 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:49.564999 2561 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5-console-config\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:22:49.565278 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:49.565007 2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5-trusted-ca-bundle\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:22:49.565278 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:49.565015 2561 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5-oauth-serving-cert\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:22:50.162241 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:50.162206 2561 generic.go:356] "Generic (PLEG): container finished" podID="e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5" containerID="a512123646cfbeea74d4c342b5d92af674d744fa0b7e1b05af35e896ada70393" exitCode=0 Sep 11 18:22:50.162659 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:50.162248 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bb5b74866-mtlnv" Sep 11 18:22:50.162659 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:50.162290 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bb5b74866-mtlnv" event={"ID":"e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5","Type":"ContainerDied","Data":"a512123646cfbeea74d4c342b5d92af674d744fa0b7e1b05af35e896ada70393"} Sep 11 18:22:50.162659 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:50.162346 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bb5b74866-mtlnv" event={"ID":"e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5","Type":"ContainerDied","Data":"dd993a7fb176d91a7f32d1201d69097a978eba11d884e2409f744373795c9100"} Sep 11 18:22:50.162659 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:50.162371 2561 scope.go:117] "RemoveContainer" containerID="a512123646cfbeea74d4c342b5d92af674d744fa0b7e1b05af35e896ada70393" Sep 11 18:22:50.171637 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:50.171618 2561 scope.go:117] "RemoveContainer" containerID="a512123646cfbeea74d4c342b5d92af674d744fa0b7e1b05af35e896ada70393" Sep 11 18:22:50.171962 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:22:50.171942 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a512123646cfbeea74d4c342b5d92af674d744fa0b7e1b05af35e896ada70393\": container with ID starting with a512123646cfbeea74d4c342b5d92af674d744fa0b7e1b05af35e896ada70393 not found: ID does not exist" containerID="a512123646cfbeea74d4c342b5d92af674d744fa0b7e1b05af35e896ada70393" Sep 11 18:22:50.172029 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:50.171969 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a512123646cfbeea74d4c342b5d92af674d744fa0b7e1b05af35e896ada70393"} err="failed to get container status \"a512123646cfbeea74d4c342b5d92af674d744fa0b7e1b05af35e896ada70393\": rpc error: code = NotFound desc = could not find container \"a512123646cfbeea74d4c342b5d92af674d744fa0b7e1b05af35e896ada70393\": container with ID starting with a512123646cfbeea74d4c342b5d92af674d744fa0b7e1b05af35e896ada70393 not found: ID does not exist" Sep 11 18:22:50.185420 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:50.185388 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bb5b74866-mtlnv"] Sep 11 18:22:50.190307 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:50.190280 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-bb5b74866-mtlnv"] Sep 11 18:22:51.139009 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:51.138975 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5" path="/var/lib/kubelet/pods/e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5/volumes" Sep 11 18:22:55.761677 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:55.761636 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahm44q"] Sep 11 18:22:55.763112 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:55.763073 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5" containerName="console" Sep 11 18:22:55.763112 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:55.763111 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5" containerName="console" Sep 11 18:22:55.763302 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:55.763269 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e53d9fd4-d77d-4d4f-b12b-9d91211e5ce5" containerName="console" Sep 11 18:22:55.769047 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:55.769019 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahm44q" Sep 11 18:22:55.772048 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:55.772018 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 11 18:22:55.772194 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:55.772014 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 11 18:22:55.772876 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:55.772849 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-gfgp7\"" Sep 11 18:22:55.773300 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:55.773273 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahm44q"] Sep 11 18:22:55.825241 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:55.825203 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pv5p\" (UniqueName: \"kubernetes.io/projected/ff1f2778-558d-43aa-9402-1686c0d2df0b-kube-api-access-9pv5p\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahm44q\" (UID: \"ff1f2778-558d-43aa-9402-1686c0d2df0b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahm44q" Sep 11 18:22:55.825445 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:55.825369 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff1f2778-558d-43aa-9402-1686c0d2df0b-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahm44q\" (UID: \"ff1f2778-558d-43aa-9402-1686c0d2df0b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahm44q" Sep 11 18:22:55.825445 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:55.825406 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff1f2778-558d-43aa-9402-1686c0d2df0b-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahm44q\" (UID: \"ff1f2778-558d-43aa-9402-1686c0d2df0b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahm44q" Sep 11 18:22:55.859209 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:55.859175 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cncm68"] Sep 11 18:22:55.862742 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:55.862714 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cncm68" Sep 11 18:22:55.871697 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:55.871669 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cncm68"] Sep 11 18:22:55.926076 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:55.926016 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff1f2778-558d-43aa-9402-1686c0d2df0b-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahm44q\" (UID: \"ff1f2778-558d-43aa-9402-1686c0d2df0b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahm44q" Sep 11 18:22:55.926076 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:55.926073 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff1f2778-558d-43aa-9402-1686c0d2df0b-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahm44q\" (UID: \"ff1f2778-558d-43aa-9402-1686c0d2df0b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahm44q" Sep 11 18:22:55.926327 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:55.926112 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pv5p\" (UniqueName: \"kubernetes.io/projected/ff1f2778-558d-43aa-9402-1686c0d2df0b-kube-api-access-9pv5p\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahm44q\" (UID: \"ff1f2778-558d-43aa-9402-1686c0d2df0b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahm44q" Sep 11 18:22:55.926327 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:55.926134 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xf5d\" (UniqueName: \"kubernetes.io/projected/0e004937-02ac-438e-a071-49fc216bb02b-kube-api-access-5xf5d\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cncm68\" (UID: \"0e004937-02ac-438e-a071-49fc216bb02b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cncm68" Sep 11 18:22:55.926327 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:55.926194 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e004937-02ac-438e-a071-49fc216bb02b-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cncm68\" (UID: \"0e004937-02ac-438e-a071-49fc216bb02b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cncm68" Sep 11 18:22:55.926327 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:55.926234 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e004937-02ac-438e-a071-49fc216bb02b-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cncm68\" (UID: \"0e004937-02ac-438e-a071-49fc216bb02b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cncm68" Sep 11 18:22:55.926512 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:55.926413 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff1f2778-558d-43aa-9402-1686c0d2df0b-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahm44q\" (UID: \"ff1f2778-558d-43aa-9402-1686c0d2df0b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahm44q" Sep 11 18:22:55.926512 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:55.926431 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff1f2778-558d-43aa-9402-1686c0d2df0b-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahm44q\" (UID: \"ff1f2778-558d-43aa-9402-1686c0d2df0b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahm44q" Sep 11 18:22:55.935342 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:55.935313 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pv5p\" (UniqueName: \"kubernetes.io/projected/ff1f2778-558d-43aa-9402-1686c0d2df0b-kube-api-access-9pv5p\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahm44q\" (UID: \"ff1f2778-558d-43aa-9402-1686c0d2df0b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahm44q" Sep 11 18:22:55.958662 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:55.958626 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788wqk"] Sep 11 18:22:55.962270 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:55.962249 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788wqk" Sep 11 18:22:55.970515 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:55.970487 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788wqk"] Sep 11 18:22:56.027287 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:56.027193 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6bjn\" (UniqueName: \"kubernetes.io/projected/b7385d0f-5c2a-459f-90dd-c2b099a0006c-kube-api-access-m6bjn\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788wqk\" (UID: \"b7385d0f-5c2a-459f-90dd-c2b099a0006c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788wqk" Sep 11 18:22:56.027287 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:56.027243 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xf5d\" (UniqueName: \"kubernetes.io/projected/0e004937-02ac-438e-a071-49fc216bb02b-kube-api-access-5xf5d\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cncm68\" (UID: \"0e004937-02ac-438e-a071-49fc216bb02b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cncm68" Sep 11 18:22:56.027524 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:56.027345 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e004937-02ac-438e-a071-49fc216bb02b-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cncm68\" (UID: \"0e004937-02ac-438e-a071-49fc216bb02b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cncm68" Sep 11 18:22:56.027524 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:56.027385 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b7385d0f-5c2a-459f-90dd-c2b099a0006c-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788wqk\" (UID: \"b7385d0f-5c2a-459f-90dd-c2b099a0006c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788wqk" Sep 11 18:22:56.027524 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:56.027415 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e004937-02ac-438e-a071-49fc216bb02b-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cncm68\" (UID: \"0e004937-02ac-438e-a071-49fc216bb02b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cncm68" Sep 11 18:22:56.027524 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:56.027454 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b7385d0f-5c2a-459f-90dd-c2b099a0006c-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788wqk\" (UID: \"b7385d0f-5c2a-459f-90dd-c2b099a0006c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788wqk" Sep 11 18:22:56.027761 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:56.027744 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e004937-02ac-438e-a071-49fc216bb02b-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cncm68\" (UID: \"0e004937-02ac-438e-a071-49fc216bb02b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cncm68" Sep 11 18:22:56.027802 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:56.027777 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e004937-02ac-438e-a071-49fc216bb02b-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cncm68\" (UID: \"0e004937-02ac-438e-a071-49fc216bb02b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cncm68" Sep 11 18:22:56.035917 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:56.035894 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xf5d\" (UniqueName: \"kubernetes.io/projected/0e004937-02ac-438e-a071-49fc216bb02b-kube-api-access-5xf5d\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cncm68\" (UID: \"0e004937-02ac-438e-a071-49fc216bb02b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cncm68" Sep 11 18:22:56.062434 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:56.062398 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62hm7f"] Sep 11 18:22:56.065899 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:56.065880 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62hm7f" Sep 11 18:22:56.075983 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:56.075953 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62hm7f"] Sep 11 18:22:56.079364 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:56.079342 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahm44q" Sep 11 18:22:56.128278 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:56.128235 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfm82\" (UniqueName: \"kubernetes.io/projected/cbb89208-baa2-42cd-aafc-59f1d8ae30a4-kube-api-access-tfm82\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62hm7f\" (UID: \"cbb89208-baa2-42cd-aafc-59f1d8ae30a4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62hm7f" Sep 11 18:22:56.128449 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:56.128294 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cbb89208-baa2-42cd-aafc-59f1d8ae30a4-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62hm7f\" (UID: \"cbb89208-baa2-42cd-aafc-59f1d8ae30a4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62hm7f" Sep 11 18:22:56.128449 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:56.128329 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b7385d0f-5c2a-459f-90dd-c2b099a0006c-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788wqk\" (UID: \"b7385d0f-5c2a-459f-90dd-c2b099a0006c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788wqk" Sep 11 18:22:56.128449 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:56.128379 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b7385d0f-5c2a-459f-90dd-c2b099a0006c-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788wqk\" (UID: \"b7385d0f-5c2a-459f-90dd-c2b099a0006c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788wqk" Sep 11 18:22:56.128449 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:56.128432 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cbb89208-baa2-42cd-aafc-59f1d8ae30a4-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62hm7f\" (UID: \"cbb89208-baa2-42cd-aafc-59f1d8ae30a4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62hm7f" Sep 11 18:22:56.128685 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:56.128488 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6bjn\" (UniqueName: \"kubernetes.io/projected/b7385d0f-5c2a-459f-90dd-c2b099a0006c-kube-api-access-m6bjn\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788wqk\" (UID: \"b7385d0f-5c2a-459f-90dd-c2b099a0006c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788wqk" Sep 11 18:22:56.128738 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:56.128722 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b7385d0f-5c2a-459f-90dd-c2b099a0006c-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788wqk\" (UID: \"b7385d0f-5c2a-459f-90dd-c2b099a0006c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788wqk" Sep 11 18:22:56.128999 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:56.128972 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b7385d0f-5c2a-459f-90dd-c2b099a0006c-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788wqk\" (UID: \"b7385d0f-5c2a-459f-90dd-c2b099a0006c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788wqk" Sep 11 18:22:56.138458 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:56.138418 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6bjn\" (UniqueName: \"kubernetes.io/projected/b7385d0f-5c2a-459f-90dd-c2b099a0006c-kube-api-access-m6bjn\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788wqk\" (UID: \"b7385d0f-5c2a-459f-90dd-c2b099a0006c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788wqk" Sep 11 18:22:56.172234 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:56.172188 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cncm68" Sep 11 18:22:56.230180 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:56.230045 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfm82\" (UniqueName: \"kubernetes.io/projected/cbb89208-baa2-42cd-aafc-59f1d8ae30a4-kube-api-access-tfm82\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62hm7f\" (UID: \"cbb89208-baa2-42cd-aafc-59f1d8ae30a4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62hm7f" Sep 11 18:22:56.230180 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:56.230121 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cbb89208-baa2-42cd-aafc-59f1d8ae30a4-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62hm7f\" (UID: \"cbb89208-baa2-42cd-aafc-59f1d8ae30a4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62hm7f" Sep 11 18:22:56.230383 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:56.230214 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cbb89208-baa2-42cd-aafc-59f1d8ae30a4-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62hm7f\" (UID: \"cbb89208-baa2-42cd-aafc-59f1d8ae30a4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62hm7f" Sep 11 18:22:56.230711 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:56.230684 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cbb89208-baa2-42cd-aafc-59f1d8ae30a4-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62hm7f\" (UID: \"cbb89208-baa2-42cd-aafc-59f1d8ae30a4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62hm7f" Sep 11 18:22:56.230711 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:56.230707 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cbb89208-baa2-42cd-aafc-59f1d8ae30a4-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62hm7f\" (UID: \"cbb89208-baa2-42cd-aafc-59f1d8ae30a4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62hm7f" Sep 11 18:22:56.239722 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:56.239697 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfm82\" (UniqueName: \"kubernetes.io/projected/cbb89208-baa2-42cd-aafc-59f1d8ae30a4-kube-api-access-tfm82\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62hm7f\" (UID: \"cbb89208-baa2-42cd-aafc-59f1d8ae30a4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62hm7f" Sep 11 18:22:56.273414 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:56.273375 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788wqk" Sep 11 18:22:56.356549 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:56.356458 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahm44q"] Sep 11 18:22:56.358672 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:22:56.358645 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff1f2778_558d_43aa_9402_1686c0d2df0b.slice/crio-55a043d248bf370f8f046a7eb9af90d688c77623ed17fe916d4ed74b6c2fd35a WatchSource:0}: Error finding container 55a043d248bf370f8f046a7eb9af90d688c77623ed17fe916d4ed74b6c2fd35a: Status 404 returned error can't find the container with id 55a043d248bf370f8f046a7eb9af90d688c77623ed17fe916d4ed74b6c2fd35a Sep 11 18:22:56.376230 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:56.376209 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62hm7f" Sep 11 18:22:56.449986 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:56.449957 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cncm68"] Sep 11 18:22:56.451353 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:22:56.451320 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e004937_02ac_438e_a071_49fc216bb02b.slice/crio-c31a67e00aa945b48b15085cd6b01003c4f33546b4340525b094cfc96f08cedd WatchSource:0}: Error finding container c31a67e00aa945b48b15085cd6b01003c4f33546b4340525b094cfc96f08cedd: Status 404 returned error can't find the container with id c31a67e00aa945b48b15085cd6b01003c4f33546b4340525b094cfc96f08cedd Sep 11 18:22:56.546880 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:56.546849 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788wqk"] Sep 11 18:22:56.547946 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:22:56.547919 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7385d0f_5c2a_459f_90dd_c2b099a0006c.slice/crio-417142f30e73e0b50e5785ea23e0584eb92f7f66f648bef78ae148722d6baf4b WatchSource:0}: Error finding container 417142f30e73e0b50e5785ea23e0584eb92f7f66f648bef78ae148722d6baf4b: Status 404 returned error can't find the container with id 417142f30e73e0b50e5785ea23e0584eb92f7f66f648bef78ae148722d6baf4b Sep 11 18:22:56.653835 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:56.653809 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62hm7f"] Sep 11 18:22:56.655956 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:22:56.655934 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbb89208_baa2_42cd_aafc_59f1d8ae30a4.slice/crio-422906873c700ba3b34fb63c5d60785657119445d464114fd56bd438a8c5ae3b WatchSource:0}: Error finding container 422906873c700ba3b34fb63c5d60785657119445d464114fd56bd438a8c5ae3b: Status 404 returned error can't find the container with id 422906873c700ba3b34fb63c5d60785657119445d464114fd56bd438a8c5ae3b Sep 11 18:22:57.198000 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:57.197970 2561 generic.go:356] "Generic (PLEG): container finished" podID="b7385d0f-5c2a-459f-90dd-c2b099a0006c" containerID="53eb83ac9273fed5793decf5a268ba4b46742672868de3e805bf3ad592f3d83c" exitCode=0 Sep 11 18:22:57.198486 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:57.198044 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788wqk" event={"ID":"b7385d0f-5c2a-459f-90dd-c2b099a0006c","Type":"ContainerDied","Data":"53eb83ac9273fed5793decf5a268ba4b46742672868de3e805bf3ad592f3d83c"} Sep 11 18:22:57.198486 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:57.198118 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788wqk" event={"ID":"b7385d0f-5c2a-459f-90dd-c2b099a0006c","Type":"ContainerStarted","Data":"417142f30e73e0b50e5785ea23e0584eb92f7f66f648bef78ae148722d6baf4b"} Sep 11 18:22:57.199867 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:57.199842 2561 generic.go:356] "Generic (PLEG): container finished" podID="ff1f2778-558d-43aa-9402-1686c0d2df0b" containerID="66bedf2a91b925b4ef7c7ec32fb335c4b2e2bb2c3d1345e4d063fcb13587c02e" exitCode=0 Sep 11 18:22:57.199963 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:57.199930 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahm44q" event={"ID":"ff1f2778-558d-43aa-9402-1686c0d2df0b","Type":"ContainerDied","Data":"66bedf2a91b925b4ef7c7ec32fb335c4b2e2bb2c3d1345e4d063fcb13587c02e"} Sep 11 18:22:57.200016 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:57.199982 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahm44q" event={"ID":"ff1f2778-558d-43aa-9402-1686c0d2df0b","Type":"ContainerStarted","Data":"55a043d248bf370f8f046a7eb9af90d688c77623ed17fe916d4ed74b6c2fd35a"} Sep 11 18:22:57.201659 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:57.201634 2561 generic.go:356] "Generic (PLEG): container finished" podID="0e004937-02ac-438e-a071-49fc216bb02b" containerID="c9893afbd3d427052854f158c13ce75d5f466b85a99b8cc3a5289d2594f8f8be" exitCode=0 Sep 11 18:22:57.201758 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:57.201665 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cncm68" event={"ID":"0e004937-02ac-438e-a071-49fc216bb02b","Type":"ContainerDied","Data":"c9893afbd3d427052854f158c13ce75d5f466b85a99b8cc3a5289d2594f8f8be"} Sep 11 18:22:57.201758 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:57.201711 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cncm68" event={"ID":"0e004937-02ac-438e-a071-49fc216bb02b","Type":"ContainerStarted","Data":"c31a67e00aa945b48b15085cd6b01003c4f33546b4340525b094cfc96f08cedd"} Sep 11 18:22:57.203610 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:57.203588 2561 generic.go:356] "Generic (PLEG): container finished" podID="cbb89208-baa2-42cd-aafc-59f1d8ae30a4" containerID="8103764e7c56ab7304cd39b7206f6e4c0b9cb7b05c1ca4ea447a2613c0fe8deb" exitCode=0 Sep 11 18:22:57.203705 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:57.203651 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62hm7f" event={"ID":"cbb89208-baa2-42cd-aafc-59f1d8ae30a4","Type":"ContainerDied","Data":"8103764e7c56ab7304cd39b7206f6e4c0b9cb7b05c1ca4ea447a2613c0fe8deb"} Sep 11 18:22:57.203751 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:57.203709 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62hm7f" event={"ID":"cbb89208-baa2-42cd-aafc-59f1d8ae30a4","Type":"ContainerStarted","Data":"422906873c700ba3b34fb63c5d60785657119445d464114fd56bd438a8c5ae3b"} Sep 11 18:22:58.210673 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:58.210634 2561 generic.go:356] "Generic (PLEG): container finished" podID="0e004937-02ac-438e-a071-49fc216bb02b" containerID="87578b1b5892b8b9c8ee5f56cb3550109a64423b2d9d8dfe671f9d7abfa6afd7" exitCode=0 Sep 11 18:22:58.211075 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:58.210675 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cncm68" event={"ID":"0e004937-02ac-438e-a071-49fc216bb02b","Type":"ContainerDied","Data":"87578b1b5892b8b9c8ee5f56cb3550109a64423b2d9d8dfe671f9d7abfa6afd7"} Sep 11 18:22:58.212692 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:58.212662 2561 generic.go:356] "Generic (PLEG): container finished" podID="b7385d0f-5c2a-459f-90dd-c2b099a0006c" containerID="8d2cacac74ef3c608332fee573416250a93538a713581007cce86df7fe21025d" exitCode=0 Sep 11 18:22:58.212793 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:58.212724 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788wqk" event={"ID":"b7385d0f-5c2a-459f-90dd-c2b099a0006c","Type":"ContainerDied","Data":"8d2cacac74ef3c608332fee573416250a93538a713581007cce86df7fe21025d"} Sep 11 18:22:58.214658 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:58.214635 2561 generic.go:356] "Generic (PLEG): container finished" podID="ff1f2778-558d-43aa-9402-1686c0d2df0b" containerID="3618652cb175c7af2a45d54e3feebff7b4b01084497fee904475b729517f1815" exitCode=0 Sep 11 18:22:58.214749 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:58.214680 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahm44q" event={"ID":"ff1f2778-558d-43aa-9402-1686c0d2df0b","Type":"ContainerDied","Data":"3618652cb175c7af2a45d54e3feebff7b4b01084497fee904475b729517f1815"} Sep 11 18:22:59.221783 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:59.221735 2561 generic.go:356] "Generic (PLEG): container finished" podID="b7385d0f-5c2a-459f-90dd-c2b099a0006c" containerID="fe63131722dd394638e89375642d6444859b85c50d918fbcc69f2410daa85499" exitCode=0 Sep 11 18:22:59.222168 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:59.221807 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788wqk" event={"ID":"b7385d0f-5c2a-459f-90dd-c2b099a0006c","Type":"ContainerDied","Data":"fe63131722dd394638e89375642d6444859b85c50d918fbcc69f2410daa85499"} Sep 11 18:22:59.224026 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:59.224001 2561 generic.go:356] "Generic (PLEG): container finished" podID="ff1f2778-558d-43aa-9402-1686c0d2df0b" containerID="ceaab81f53cd34b4314040517a090de62341c3d7e8e0a284c0bd54a96b5c3f23" exitCode=0 Sep 11 18:22:59.224182 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:59.224081 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahm44q" event={"ID":"ff1f2778-558d-43aa-9402-1686c0d2df0b","Type":"ContainerDied","Data":"ceaab81f53cd34b4314040517a090de62341c3d7e8e0a284c0bd54a96b5c3f23"} Sep 11 18:22:59.226128 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:59.226108 2561 generic.go:356] "Generic (PLEG): container finished" podID="0e004937-02ac-438e-a071-49fc216bb02b" containerID="c830d1caeb345e3a7220126dc33aa1a5f55439871560e4f1a8f0f5093f09853b" exitCode=0 Sep 11 18:22:59.226245 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:59.226184 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cncm68" event={"ID":"0e004937-02ac-438e-a071-49fc216bb02b","Type":"ContainerDied","Data":"c830d1caeb345e3a7220126dc33aa1a5f55439871560e4f1a8f0f5093f09853b"} Sep 11 18:22:59.227870 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:59.227852 2561 generic.go:356] "Generic (PLEG): container finished" podID="cbb89208-baa2-42cd-aafc-59f1d8ae30a4" containerID="56176dd0717050b7f3de7e4fb01b69837a34693ba3f4145905a3c7d22c087e79" exitCode=0 Sep 11 18:22:59.227940 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:22:59.227898 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62hm7f" event={"ID":"cbb89208-baa2-42cd-aafc-59f1d8ae30a4","Type":"ContainerDied","Data":"56176dd0717050b7f3de7e4fb01b69837a34693ba3f4145905a3c7d22c087e79"} Sep 11 18:23:00.235590 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:00.235558 2561 generic.go:356] "Generic (PLEG): container finished" podID="cbb89208-baa2-42cd-aafc-59f1d8ae30a4" containerID="de65acd5019b3d32886ba60329e0b8cad293563bbb584c7f220654323697b438" exitCode=0 Sep 11 18:23:00.236044 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:00.235638 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62hm7f" event={"ID":"cbb89208-baa2-42cd-aafc-59f1d8ae30a4","Type":"ContainerDied","Data":"de65acd5019b3d32886ba60329e0b8cad293563bbb584c7f220654323697b438"} Sep 11 18:23:00.394573 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:00.394544 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cncm68" Sep 11 18:23:00.435435 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:00.435409 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788wqk" Sep 11 18:23:00.438849 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:00.438825 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahm44q" Sep 11 18:23:00.472306 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:00.472269 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b7385d0f-5c2a-459f-90dd-c2b099a0006c-util\") pod \"b7385d0f-5c2a-459f-90dd-c2b099a0006c\" (UID: \"b7385d0f-5c2a-459f-90dd-c2b099a0006c\") " Sep 11 18:23:00.472468 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:00.472322 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff1f2778-558d-43aa-9402-1686c0d2df0b-bundle\") pod \"ff1f2778-558d-43aa-9402-1686c0d2df0b\" (UID: \"ff1f2778-558d-43aa-9402-1686c0d2df0b\") " Sep 11 18:23:00.472468 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:00.472359 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff1f2778-558d-43aa-9402-1686c0d2df0b-util\") pod \"ff1f2778-558d-43aa-9402-1686c0d2df0b\" (UID: \"ff1f2778-558d-43aa-9402-1686c0d2df0b\") " Sep 11 18:23:00.472468 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:00.472406 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e004937-02ac-438e-a071-49fc216bb02b-bundle\") pod \"0e004937-02ac-438e-a071-49fc216bb02b\" (UID: \"0e004937-02ac-438e-a071-49fc216bb02b\") " Sep 11 18:23:00.472468 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:00.472440 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xf5d\" (UniqueName: \"kubernetes.io/projected/0e004937-02ac-438e-a071-49fc216bb02b-kube-api-access-5xf5d\") pod \"0e004937-02ac-438e-a071-49fc216bb02b\" (UID: \"0e004937-02ac-438e-a071-49fc216bb02b\") " Sep 11 18:23:00.472468 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:00.472466 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b7385d0f-5c2a-459f-90dd-c2b099a0006c-bundle\") pod \"b7385d0f-5c2a-459f-90dd-c2b099a0006c\" (UID: \"b7385d0f-5c2a-459f-90dd-c2b099a0006c\") " Sep 11 18:23:00.472728 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:00.472512 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e004937-02ac-438e-a071-49fc216bb02b-util\") pod \"0e004937-02ac-438e-a071-49fc216bb02b\" (UID: \"0e004937-02ac-438e-a071-49fc216bb02b\") " Sep 11 18:23:00.472728 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:00.472548 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9pv5p\" (UniqueName: \"kubernetes.io/projected/ff1f2778-558d-43aa-9402-1686c0d2df0b-kube-api-access-9pv5p\") pod \"ff1f2778-558d-43aa-9402-1686c0d2df0b\" (UID: \"ff1f2778-558d-43aa-9402-1686c0d2df0b\") " Sep 11 18:23:00.472728 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:00.472618 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6bjn\" (UniqueName: \"kubernetes.io/projected/b7385d0f-5c2a-459f-90dd-c2b099a0006c-kube-api-access-m6bjn\") pod \"b7385d0f-5c2a-459f-90dd-c2b099a0006c\" (UID: \"b7385d0f-5c2a-459f-90dd-c2b099a0006c\") " Sep 11 18:23:00.475589 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:00.475547 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7385d0f-5c2a-459f-90dd-c2b099a0006c-bundle" (OuterVolumeSpecName: "bundle") pod "b7385d0f-5c2a-459f-90dd-c2b099a0006c" (UID: "b7385d0f-5c2a-459f-90dd-c2b099a0006c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:23:00.476508 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:00.476318 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e004937-02ac-438e-a071-49fc216bb02b-bundle" (OuterVolumeSpecName: "bundle") pod "0e004937-02ac-438e-a071-49fc216bb02b" (UID: "0e004937-02ac-438e-a071-49fc216bb02b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:23:00.476508 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:00.476386 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff1f2778-558d-43aa-9402-1686c0d2df0b-bundle" (OuterVolumeSpecName: "bundle") pod "ff1f2778-558d-43aa-9402-1686c0d2df0b" (UID: "ff1f2778-558d-43aa-9402-1686c0d2df0b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:23:00.476508 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:00.476477 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7385d0f-5c2a-459f-90dd-c2b099a0006c-kube-api-access-m6bjn" (OuterVolumeSpecName: "kube-api-access-m6bjn") pod "b7385d0f-5c2a-459f-90dd-c2b099a0006c" (UID: "b7385d0f-5c2a-459f-90dd-c2b099a0006c"). InnerVolumeSpecName "kube-api-access-m6bjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:23:00.479024 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:00.478923 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e004937-02ac-438e-a071-49fc216bb02b-kube-api-access-5xf5d" (OuterVolumeSpecName: "kube-api-access-5xf5d") pod "0e004937-02ac-438e-a071-49fc216bb02b" (UID: "0e004937-02ac-438e-a071-49fc216bb02b"). InnerVolumeSpecName "kube-api-access-5xf5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:23:00.479167 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:00.479077 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff1f2778-558d-43aa-9402-1686c0d2df0b-kube-api-access-9pv5p" (OuterVolumeSpecName: "kube-api-access-9pv5p") pod "ff1f2778-558d-43aa-9402-1686c0d2df0b" (UID: "ff1f2778-558d-43aa-9402-1686c0d2df0b"). InnerVolumeSpecName "kube-api-access-9pv5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:23:00.481617 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:00.481593 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7385d0f-5c2a-459f-90dd-c2b099a0006c-util" (OuterVolumeSpecName: "util") pod "b7385d0f-5c2a-459f-90dd-c2b099a0006c" (UID: "b7385d0f-5c2a-459f-90dd-c2b099a0006c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:23:00.482496 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:00.482468 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e004937-02ac-438e-a071-49fc216bb02b-util" (OuterVolumeSpecName: "util") pod "0e004937-02ac-438e-a071-49fc216bb02b" (UID: "0e004937-02ac-438e-a071-49fc216bb02b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:23:00.483865 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:00.483841 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff1f2778-558d-43aa-9402-1686c0d2df0b-util" (OuterVolumeSpecName: "util") pod "ff1f2778-558d-43aa-9402-1686c0d2df0b" (UID: "ff1f2778-558d-43aa-9402-1686c0d2df0b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:23:00.574128 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:00.574002 2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b7385d0f-5c2a-459f-90dd-c2b099a0006c-util\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:23:00.574128 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:00.574034 2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff1f2778-558d-43aa-9402-1686c0d2df0b-bundle\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:23:00.574128 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:00.574047 2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff1f2778-558d-43aa-9402-1686c0d2df0b-util\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:23:00.574128 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:00.574072 2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e004937-02ac-438e-a071-49fc216bb02b-bundle\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:23:00.574128 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:00.574085 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xf5d\" (UniqueName: \"kubernetes.io/projected/0e004937-02ac-438e-a071-49fc216bb02b-kube-api-access-5xf5d\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:23:00.574128 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:00.574098 2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b7385d0f-5c2a-459f-90dd-c2b099a0006c-bundle\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:23:00.574128 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:00.574109 2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e004937-02ac-438e-a071-49fc216bb02b-util\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:23:00.574128 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:00.574122 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pv5p\" (UniqueName: \"kubernetes.io/projected/ff1f2778-558d-43aa-9402-1686c0d2df0b-kube-api-access-9pv5p\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:23:00.574128 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:00.574134 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6bjn\" (UniqueName: \"kubernetes.io/projected/b7385d0f-5c2a-459f-90dd-c2b099a0006c-kube-api-access-m6bjn\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:23:01.242821 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:01.242794 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cncm68" Sep 11 18:23:01.242821 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:01.242813 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cncm68" event={"ID":"0e004937-02ac-438e-a071-49fc216bb02b","Type":"ContainerDied","Data":"c31a67e00aa945b48b15085cd6b01003c4f33546b4340525b094cfc96f08cedd"} Sep 11 18:23:01.243258 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:01.242864 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c31a67e00aa945b48b15085cd6b01003c4f33546b4340525b094cfc96f08cedd" Sep 11 18:23:01.245218 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:01.245185 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788wqk" event={"ID":"b7385d0f-5c2a-459f-90dd-c2b099a0006c","Type":"ContainerDied","Data":"417142f30e73e0b50e5785ea23e0584eb92f7f66f648bef78ae148722d6baf4b"} Sep 11 18:23:01.245338 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:01.245242 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="417142f30e73e0b50e5785ea23e0584eb92f7f66f648bef78ae148722d6baf4b" Sep 11 18:23:01.245338 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:01.245195 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d788wqk" Sep 11 18:23:01.247450 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:01.247432 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahm44q" Sep 11 18:23:01.247541 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:01.247447 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahm44q" event={"ID":"ff1f2778-558d-43aa-9402-1686c0d2df0b","Type":"ContainerDied","Data":"55a043d248bf370f8f046a7eb9af90d688c77623ed17fe916d4ed74b6c2fd35a"} Sep 11 18:23:01.247541 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:01.247486 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55a043d248bf370f8f046a7eb9af90d688c77623ed17fe916d4ed74b6c2fd35a" Sep 11 18:23:01.375942 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:01.375876 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62hm7f" Sep 11 18:23:01.480984 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:01.480948 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cbb89208-baa2-42cd-aafc-59f1d8ae30a4-util\") pod \"cbb89208-baa2-42cd-aafc-59f1d8ae30a4\" (UID: \"cbb89208-baa2-42cd-aafc-59f1d8ae30a4\") " Sep 11 18:23:01.481158 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:01.481083 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tfm82\" (UniqueName: \"kubernetes.io/projected/cbb89208-baa2-42cd-aafc-59f1d8ae30a4-kube-api-access-tfm82\") pod \"cbb89208-baa2-42cd-aafc-59f1d8ae30a4\" (UID: \"cbb89208-baa2-42cd-aafc-59f1d8ae30a4\") " Sep 11 18:23:01.481158 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:01.481114 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cbb89208-baa2-42cd-aafc-59f1d8ae30a4-bundle\") pod \"cbb89208-baa2-42cd-aafc-59f1d8ae30a4\" (UID: \"cbb89208-baa2-42cd-aafc-59f1d8ae30a4\") " Sep 11 18:23:01.481647 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:01.481620 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbb89208-baa2-42cd-aafc-59f1d8ae30a4-bundle" (OuterVolumeSpecName: "bundle") pod "cbb89208-baa2-42cd-aafc-59f1d8ae30a4" (UID: "cbb89208-baa2-42cd-aafc-59f1d8ae30a4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:23:01.483155 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:01.483130 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbb89208-baa2-42cd-aafc-59f1d8ae30a4-kube-api-access-tfm82" (OuterVolumeSpecName: "kube-api-access-tfm82") pod "cbb89208-baa2-42cd-aafc-59f1d8ae30a4" (UID: "cbb89208-baa2-42cd-aafc-59f1d8ae30a4"). InnerVolumeSpecName "kube-api-access-tfm82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:23:01.486579 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:01.486547 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbb89208-baa2-42cd-aafc-59f1d8ae30a4-util" (OuterVolumeSpecName: "util") pod "cbb89208-baa2-42cd-aafc-59f1d8ae30a4" (UID: "cbb89208-baa2-42cd-aafc-59f1d8ae30a4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:23:01.582562 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:01.582522 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfm82\" (UniqueName: \"kubernetes.io/projected/cbb89208-baa2-42cd-aafc-59f1d8ae30a4-kube-api-access-tfm82\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:23:01.582562 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:01.582555 2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cbb89208-baa2-42cd-aafc-59f1d8ae30a4-bundle\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:23:01.582562 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:01.582566 2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cbb89208-baa2-42cd-aafc-59f1d8ae30a4-util\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:23:02.254759 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:02.254717 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62hm7f" event={"ID":"cbb89208-baa2-42cd-aafc-59f1d8ae30a4","Type":"ContainerDied","Data":"422906873c700ba3b34fb63c5d60785657119445d464114fd56bd438a8c5ae3b"} Sep 11 18:23:02.254759 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:02.254743 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62hm7f" Sep 11 18:23:02.254759 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:02.254765 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="422906873c700ba3b34fb63c5d60785657119445d464114fd56bd438a8c5ae3b" Sep 11 18:23:07.807624 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:07.807586 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-qc2xg"] Sep 11 18:23:07.808030 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:07.807962 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff1f2778-558d-43aa-9402-1686c0d2df0b" containerName="extract" Sep 11 18:23:07.808030 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:07.807976 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff1f2778-558d-43aa-9402-1686c0d2df0b" containerName="extract" Sep 11 18:23:07.808030 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:07.807990 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e004937-02ac-438e-a071-49fc216bb02b" containerName="pull" Sep 11 18:23:07.808030 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:07.807997 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e004937-02ac-438e-a071-49fc216bb02b" containerName="pull" Sep 11 18:23:07.808030 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:07.808008 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbb89208-baa2-42cd-aafc-59f1d8ae30a4" containerName="extract" Sep 11 18:23:07.808030 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:07.808016 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbb89208-baa2-42cd-aafc-59f1d8ae30a4" containerName="extract" Sep 11 18:23:07.808030 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:07.808030 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbb89208-baa2-42cd-aafc-59f1d8ae30a4" containerName="util" Sep 11 18:23:07.808030 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:07.808035 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbb89208-baa2-42cd-aafc-59f1d8ae30a4" containerName="util" Sep 11 18:23:07.808291 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:07.808041 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7385d0f-5c2a-459f-90dd-c2b099a0006c" containerName="pull" Sep 11 18:23:07.808291 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:07.808046 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7385d0f-5c2a-459f-90dd-c2b099a0006c" containerName="pull" Sep 11 18:23:07.808291 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:07.808064 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff1f2778-558d-43aa-9402-1686c0d2df0b" containerName="pull" Sep 11 18:23:07.808291 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:07.808070 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff1f2778-558d-43aa-9402-1686c0d2df0b" containerName="pull" Sep 11 18:23:07.808291 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:07.808081 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7385d0f-5c2a-459f-90dd-c2b099a0006c" containerName="extract" Sep 11 18:23:07.808291 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:07.808089 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7385d0f-5c2a-459f-90dd-c2b099a0006c" containerName="extract" Sep 11 18:23:07.808291 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:07.808100 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e004937-02ac-438e-a071-49fc216bb02b" containerName="util" Sep 11 18:23:07.808291 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:07.808106 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e004937-02ac-438e-a071-49fc216bb02b" containerName="util" Sep 11 18:23:07.808291 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:07.808115 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e004937-02ac-438e-a071-49fc216bb02b" containerName="extract" Sep 11 18:23:07.808291 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:07.808121 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e004937-02ac-438e-a071-49fc216bb02b" containerName="extract" Sep 11 18:23:07.808291 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:07.808126 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff1f2778-558d-43aa-9402-1686c0d2df0b" containerName="util" Sep 11 18:23:07.808291 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:07.808131 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff1f2778-558d-43aa-9402-1686c0d2df0b" containerName="util" Sep 11 18:23:07.808291 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:07.808137 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7385d0f-5c2a-459f-90dd-c2b099a0006c" containerName="util" Sep 11 18:23:07.808291 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:07.808142 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7385d0f-5c2a-459f-90dd-c2b099a0006c" containerName="util" Sep 11 18:23:07.808291 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:07.808149 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbb89208-baa2-42cd-aafc-59f1d8ae30a4" containerName="pull" Sep 11 18:23:07.808291 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:07.808154 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbb89208-baa2-42cd-aafc-59f1d8ae30a4" containerName="pull" Sep 11 18:23:07.808291 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:07.808219 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff1f2778-558d-43aa-9402-1686c0d2df0b" containerName="extract" Sep 11 18:23:07.808291 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:07.808227 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbb89208-baa2-42cd-aafc-59f1d8ae30a4" containerName="extract" Sep 11 18:23:07.808291 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:07.808236 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7385d0f-5c2a-459f-90dd-c2b099a0006c" containerName="extract" Sep 11 18:23:07.808291 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:07.808245 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e004937-02ac-438e-a071-49fc216bb02b" containerName="extract" Sep 11 18:23:07.811578 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:07.811560 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qc2xg" Sep 11 18:23:07.814042 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:07.814013 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 11 18:23:07.814166 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:07.814016 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-djhbt\"" Sep 11 18:23:07.814704 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:07.814680 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 11 18:23:07.822497 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:07.822469 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-qc2xg"] Sep 11 18:23:07.939673 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:07.939621 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v6pg\" (UniqueName: \"kubernetes.io/projected/fe464dd0-84f3-4d43-b74e-f44ca0d4ebf2-kube-api-access-4v6pg\") pod \"limitador-operator-controller-manager-74495c69f8-qc2xg\" (UID: \"fe464dd0-84f3-4d43-b74e-f44ca0d4ebf2\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qc2xg" Sep 11 18:23:08.041194 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:08.041139 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v6pg\" (UniqueName: \"kubernetes.io/projected/fe464dd0-84f3-4d43-b74e-f44ca0d4ebf2-kube-api-access-4v6pg\") pod \"limitador-operator-controller-manager-74495c69f8-qc2xg\" (UID: \"fe464dd0-84f3-4d43-b74e-f44ca0d4ebf2\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qc2xg" Sep 11 18:23:08.051483 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:08.051451 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v6pg\" (UniqueName: \"kubernetes.io/projected/fe464dd0-84f3-4d43-b74e-f44ca0d4ebf2-kube-api-access-4v6pg\") pod \"limitador-operator-controller-manager-74495c69f8-qc2xg\" (UID: \"fe464dd0-84f3-4d43-b74e-f44ca0d4ebf2\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qc2xg" Sep 11 18:23:08.122334 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:08.122251 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qc2xg" Sep 11 18:23:08.400169 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:08.400137 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-qc2xg"] Sep 11 18:23:08.402288 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:23:08.402255 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe464dd0_84f3_4d43_b74e_f44ca0d4ebf2.slice/crio-aa57b3167996c1877b85b7dd4739c4d55e0ade69986308cc896eedfb6a060022 WatchSource:0}: Error finding container aa57b3167996c1877b85b7dd4739c4d55e0ade69986308cc896eedfb6a060022: Status 404 returned error can't find the container with id aa57b3167996c1877b85b7dd4739c4d55e0ade69986308cc896eedfb6a060022 Sep 11 18:23:09.303068 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:09.303011 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qc2xg" event={"ID":"fe464dd0-84f3-4d43-b74e-f44ca0d4ebf2","Type":"ContainerStarted","Data":"aa57b3167996c1877b85b7dd4739c4d55e0ade69986308cc896eedfb6a060022"} Sep 11 18:23:10.242447 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:10.242408 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-dvxrr"] Sep 11 18:23:10.251796 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:10.251762 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-dvxrr" Sep 11 18:23:10.256488 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:10.256410 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-88584\"" Sep 11 18:23:10.259416 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:10.259381 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-dvxrr"] Sep 11 18:23:10.369142 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:10.369092 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9cqx\" (UniqueName: \"kubernetes.io/projected/dc27b87c-4bf6-49de-9a7f-a74ee17929f3-kube-api-access-b9cqx\") pod \"authorino-operator-76b478bd78-dvxrr\" (UID: \"dc27b87c-4bf6-49de-9a7f-a74ee17929f3\") " pod="kuadrant-system/authorino-operator-76b478bd78-dvxrr" Sep 11 18:23:10.470295 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:10.470254 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9cqx\" (UniqueName: \"kubernetes.io/projected/dc27b87c-4bf6-49de-9a7f-a74ee17929f3-kube-api-access-b9cqx\") pod \"authorino-operator-76b478bd78-dvxrr\" (UID: \"dc27b87c-4bf6-49de-9a7f-a74ee17929f3\") " pod="kuadrant-system/authorino-operator-76b478bd78-dvxrr" Sep 11 18:23:10.481644 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:10.481606 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9cqx\" (UniqueName: \"kubernetes.io/projected/dc27b87c-4bf6-49de-9a7f-a74ee17929f3-kube-api-access-b9cqx\") pod \"authorino-operator-76b478bd78-dvxrr\" (UID: \"dc27b87c-4bf6-49de-9a7f-a74ee17929f3\") " pod="kuadrant-system/authorino-operator-76b478bd78-dvxrr" Sep 11 18:23:10.566296 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:10.566223 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-dvxrr" Sep 11 18:23:11.116307 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:11.116278 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-dvxrr"] Sep 11 18:23:11.117313 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:23:11.117283 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc27b87c_4bf6_49de_9a7f_a74ee17929f3.slice/crio-4dd67bffda09bd2fca01211020f40773c3bca91d4a8ae29988b7fc65b827107e WatchSource:0}: Error finding container 4dd67bffda09bd2fca01211020f40773c3bca91d4a8ae29988b7fc65b827107e: Status 404 returned error can't find the container with id 4dd67bffda09bd2fca01211020f40773c3bca91d4a8ae29988b7fc65b827107e Sep 11 18:23:11.319347 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:11.319303 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qc2xg" event={"ID":"fe464dd0-84f3-4d43-b74e-f44ca0d4ebf2","Type":"ContainerStarted","Data":"1f6d24e3ed77579184cacee0f0bf65ed83d3353c13fa5fc9dff659c2684ae80e"} Sep 11 18:23:11.319506 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:11.319419 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qc2xg" Sep 11 18:23:11.320799 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:11.320773 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-dvxrr" event={"ID":"dc27b87c-4bf6-49de-9a7f-a74ee17929f3","Type":"ContainerStarted","Data":"4dd67bffda09bd2fca01211020f40773c3bca91d4a8ae29988b7fc65b827107e"} Sep 11 18:23:11.337186 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:11.337140 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qc2xg" podStartSLOduration=1.826767615 podStartE2EDuration="4.337122177s" totalImagesPullingTime="2.510354562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:23:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:23:11.336575181 +0000 UTC m=+472.833592937" watchObservedRunningTime="2026-09-11 18:23:11.337122177 +0000 UTC m=+472.834139945" Sep 11 18:23:14.650079 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:14.650017 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rdn5x"] Sep 11 18:23:14.653735 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:14.653713 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rdn5x" Sep 11 18:23:14.656122 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:14.656095 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-j46f2\"" Sep 11 18:23:14.665433 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:14.665405 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rdn5x"] Sep 11 18:23:14.812946 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:14.812904 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/012e75f6-47cb-4112-bc94-63c492558cab-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-rdn5x\" (UID: \"012e75f6-47cb-4112-bc94-63c492558cab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rdn5x" Sep 11 18:23:14.813186 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:14.812959 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqmvj\" (UniqueName: \"kubernetes.io/projected/012e75f6-47cb-4112-bc94-63c492558cab-kube-api-access-xqmvj\") pod \"kuadrant-operator-controller-manager-76f8f686bf-rdn5x\" (UID: \"012e75f6-47cb-4112-bc94-63c492558cab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rdn5x" Sep 11 18:23:14.913899 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:14.913860 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqmvj\" (UniqueName: \"kubernetes.io/projected/012e75f6-47cb-4112-bc94-63c492558cab-kube-api-access-xqmvj\") pod \"kuadrant-operator-controller-manager-76f8f686bf-rdn5x\" (UID: \"012e75f6-47cb-4112-bc94-63c492558cab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rdn5x" Sep 11 18:23:14.914121 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:14.913981 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/012e75f6-47cb-4112-bc94-63c492558cab-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-rdn5x\" (UID: \"012e75f6-47cb-4112-bc94-63c492558cab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rdn5x" Sep 11 18:23:14.914405 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:14.914382 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/012e75f6-47cb-4112-bc94-63c492558cab-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-rdn5x\" (UID: \"012e75f6-47cb-4112-bc94-63c492558cab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rdn5x" Sep 11 18:23:14.926666 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:14.926632 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqmvj\" (UniqueName: \"kubernetes.io/projected/012e75f6-47cb-4112-bc94-63c492558cab-kube-api-access-xqmvj\") pod \"kuadrant-operator-controller-manager-76f8f686bf-rdn5x\" (UID: \"012e75f6-47cb-4112-bc94-63c492558cab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rdn5x" Sep 11 18:23:14.965622 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:14.965579 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rdn5x" Sep 11 18:23:15.241095 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:15.241064 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rdn5x"] Sep 11 18:23:15.242604 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:23:15.242575 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod012e75f6_47cb_4112_bc94_63c492558cab.slice/crio-2a9d3b1a8a3b5e620f8173425da87bf86a6327616e05b238de666793a8a98a6c WatchSource:0}: Error finding container 2a9d3b1a8a3b5e620f8173425da87bf86a6327616e05b238de666793a8a98a6c: Status 404 returned error can't find the container with id 2a9d3b1a8a3b5e620f8173425da87bf86a6327616e05b238de666793a8a98a6c Sep 11 18:23:15.342753 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:15.342715 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rdn5x" event={"ID":"012e75f6-47cb-4112-bc94-63c492558cab","Type":"ContainerStarted","Data":"2a9d3b1a8a3b5e620f8173425da87bf86a6327616e05b238de666793a8a98a6c"} Sep 11 18:23:19.371417 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:19.371295 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rdn5x" event={"ID":"012e75f6-47cb-4112-bc94-63c492558cab","Type":"ContainerStarted","Data":"68c83127e3338d0f70d9811f65696371d20f8a0e6621c6ab07e62e98045caf6d"} Sep 11 18:23:19.371860 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:19.371485 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rdn5x" Sep 11 18:23:19.392476 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:19.392412 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rdn5x" podStartSLOduration=1.531921832 podStartE2EDuration="5.392391445s" totalImagesPullingTime="3.860469613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:23:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:23:19.391285297 +0000 UTC m=+480.888303057" watchObservedRunningTime="2026-09-11 18:23:19.392391445 +0000 UTC m=+480.889409200" Sep 11 18:23:20.378241 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:20.378203 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-dvxrr" event={"ID":"dc27b87c-4bf6-49de-9a7f-a74ee17929f3","Type":"ContainerStarted","Data":"575a65316fe1a81ca211e97f89b512768aa57a533628d72c5c696ebf0258fe93"} Sep 11 18:23:20.378646 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:20.378310 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-dvxrr" Sep 11 18:23:20.396875 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:20.396814 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-dvxrr" podStartSLOduration=1.692030691 podStartE2EDuration="10.396796633s" totalImagesPullingTime="8.704765942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:23:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:23:20.394808127 +0000 UTC m=+481.891825881" watchObservedRunningTime="2026-09-11 18:23:20.396796633 +0000 UTC m=+481.893814397" Sep 11 18:23:22.328571 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:22.328536 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qc2xg" Sep 11 18:23:30.380467 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:30.380435 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rdn5x" Sep 11 18:23:31.385902 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:31.385868 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-dvxrr" Sep 11 18:23:32.282766 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:32.282724 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6zfkv"] Sep 11 18:23:32.287781 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:32.287763 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6zfkv" Sep 11 18:23:32.309229 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:32.309197 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6zfkv"] Sep 11 18:23:32.374006 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:32.373955 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b487f8fd-50ca-453d-8e2a-3203520aa888-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-6zfkv\" (UID: \"b487f8fd-50ca-453d-8e2a-3203520aa888\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6zfkv" Sep 11 18:23:32.374202 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:32.374049 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4h6p\" (UniqueName: \"kubernetes.io/projected/b487f8fd-50ca-453d-8e2a-3203520aa888-kube-api-access-b4h6p\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-6zfkv\" (UID: \"b487f8fd-50ca-453d-8e2a-3203520aa888\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6zfkv" Sep 11 18:23:32.454513 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:32.454476 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rdn5x"] Sep 11 18:23:32.454896 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:32.454665 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rdn5x" podUID="012e75f6-47cb-4112-bc94-63c492558cab" containerName="manager" containerID="cri-o://68c83127e3338d0f70d9811f65696371d20f8a0e6621c6ab07e62e98045caf6d" gracePeriod=2 Sep 11 18:23:32.475394 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:32.475361 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b487f8fd-50ca-453d-8e2a-3203520aa888-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-6zfkv\" (UID: \"b487f8fd-50ca-453d-8e2a-3203520aa888\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6zfkv" Sep 11 18:23:32.475487 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:32.475451 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4h6p\" (UniqueName: \"kubernetes.io/projected/b487f8fd-50ca-453d-8e2a-3203520aa888-kube-api-access-b4h6p\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-6zfkv\" (UID: \"b487f8fd-50ca-453d-8e2a-3203520aa888\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6zfkv" Sep 11 18:23:32.475807 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:32.475773 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b487f8fd-50ca-453d-8e2a-3203520aa888-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-6zfkv\" (UID: \"b487f8fd-50ca-453d-8e2a-3203520aa888\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6zfkv" Sep 11 18:23:32.475947 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:32.475929 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rdn5x"] Sep 11 18:23:32.484339 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:32.484314 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6zfkv"] Sep 11 18:23:32.484625 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:23:32.484604 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-b4h6p], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6zfkv" podUID="b487f8fd-50ca-453d-8e2a-3203520aa888" Sep 11 18:23:32.487432 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:23:32.487396 2561 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"012e75f6-47cb-4112-bc94-63c492558cab\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-rdn5x\": pods \"kuadrant-operator-controller-manager-76f8f686bf-rdn5x\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rdn5x" Sep 11 18:23:32.497151 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:32.497123 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4h6p\" (UniqueName: \"kubernetes.io/projected/b487f8fd-50ca-453d-8e2a-3203520aa888-kube-api-access-b4h6p\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-6zfkv\" (UID: \"b487f8fd-50ca-453d-8e2a-3203520aa888\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6zfkv" Sep 11 18:23:32.498903 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:32.498875 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6zfkv"] Sep 11 18:23:32.509234 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:32.509206 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5r2mj"] Sep 11 18:23:32.509760 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:32.509728 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="012e75f6-47cb-4112-bc94-63c492558cab" containerName="manager" Sep 11 18:23:32.509760 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:32.509754 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="012e75f6-47cb-4112-bc94-63c492558cab" containerName="manager" Sep 11 18:23:32.509915 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:32.509863 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="012e75f6-47cb-4112-bc94-63c492558cab" containerName="manager" Sep 11 18:23:32.513389 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:32.513354 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6zfkv"] Sep 11 18:23:32.513507 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:32.513487 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5r2mj" Sep 11 18:23:32.515940 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:23:32.515906 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-rdn5x\" is forbidden: User \"system:node:ip-10-0-138-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-128.ec2.internal' and this object" podUID="012e75f6-47cb-4112-bc94-63c492558cab" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rdn5x" Sep 11 18:23:32.534145 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:32.534047 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5r2mj"] Sep 11 18:23:32.544208 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:32.544177 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-qc2xg"] Sep 11 18:23:32.544479 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:32.544439 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qc2xg" podUID="fe464dd0-84f3-4d43-b74e-f44ca0d4ebf2" containerName="manager" containerID="cri-o://1f6d24e3ed77579184cacee0f0bf65ed83d3353c13fa5fc9dff659c2684ae80e" gracePeriod=2 Sep 11 18:23:32.547134 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:32.547104 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-qc2xg"] Sep 11 18:23:32.555069 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:32.555028 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lsktr"] Sep 11 18:23:32.555448 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:32.555432 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe464dd0-84f3-4d43-b74e-f44ca0d4ebf2" containerName="manager" Sep 11 18:23:32.555517 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:32.555449 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe464dd0-84f3-4d43-b74e-f44ca0d4ebf2" containerName="manager" Sep 11 18:23:32.555517 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:32.555506 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe464dd0-84f3-4d43-b74e-f44ca0d4ebf2" containerName="manager" Sep 11 18:23:32.559675 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:32.559651 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lsktr" Sep 11 18:23:32.559832 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:23:32.559782 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-rdn5x\" is forbidden: User \"system:node:ip-10-0-138-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-128.ec2.internal' and this object" podUID="012e75f6-47cb-4112-bc94-63c492558cab" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rdn5x" Sep 11 18:23:32.561976 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:23:32.561936 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-qc2xg\" is forbidden: User \"system:node:ip-10-0-138-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-128.ec2.internal' and this object" podUID="fe464dd0-84f3-4d43-b74e-f44ca0d4ebf2" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qc2xg" Sep 11 18:23:32.567683 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:23:32.567654 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-qc2xg\" is forbidden: User \"system:node:ip-10-0-138-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-128.ec2.internal' and this object" podUID="fe464dd0-84f3-4d43-b74e-f44ca0d4ebf2" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qc2xg" Sep 11 18:23:32.576898 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:32.576873 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scw9x\" (UniqueName: \"kubernetes.io/projected/fce4d870-b756-43ef-b375-3b092192f03c-kube-api-access-scw9x\") pod \"kuadrant-operator-controller-manager-76f8f686bf-5r2mj\" (UID: \"fce4d870-b756-43ef-b375-3b092192f03c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5r2mj" Sep 11 18:23:32.576990 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:32.576929 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fce4d870-b756-43ef-b375-3b092192f03c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-5r2mj\" (UID: \"fce4d870-b756-43ef-b375-3b092192f03c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5r2mj" Sep 11 18:23:32.584938 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:32.584913 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lsktr"] Sep 11 18:23:32.599152 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:32.599114 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-79pgz"] Sep 11 18:23:32.603411 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:32.603380 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-79pgz" Sep 11 18:23:32.618246 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:23:32.618212 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-rdn5x\" is forbidden: User \"system:node:ip-10-0-138-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-128.ec2.internal' and this object" podUID="012e75f6-47cb-4112-bc94-63c492558cab" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rdn5x" Sep 11 18:23:32.620847 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:23:32.620565 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-qc2xg\" is forbidden: User \"system:node:ip-10-0-138-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-128.ec2.internal' and this object" podUID="fe464dd0-84f3-4d43-b74e-f44ca0d4ebf2" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qc2xg" Sep 11 18:23:32.624253 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:32.624226 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-79pgz"] Sep 11 18:23:32.658336 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:23:32.658302 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-rdn5x\" is forbidden: User \"system:node:ip-10-0-138-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-128.ec2.internal' and this object" podUID="012e75f6-47cb-4112-bc94-63c492558cab" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rdn5x" Sep 11 18:23:32.678082 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:32.677967 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scw9x\" (UniqueName: \"kubernetes.io/projected/fce4d870-b756-43ef-b375-3b092192f03c-kube-api-access-scw9x\") pod \"kuadrant-operator-controller-manager-76f8f686bf-5r2mj\" (UID: \"fce4d870-b756-43ef-b375-3b092192f03c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5r2mj" Sep 11 18:23:32.678082 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:32.678044 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0059eaea-499d-499c-98d6-9547fc02722d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-lsktr\" (UID: \"0059eaea-499d-499c-98d6-9547fc02722d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lsktr" Sep 11 18:23:32.678273 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:32.678206 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fce4d870-b756-43ef-b375-3b092192f03c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-5r2mj\" (UID: \"fce4d870-b756-43ef-b375-3b092192f03c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5r2mj" Sep 11 18:23:32.678273 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:32.678245 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lzlp\" (UniqueName: \"kubernetes.io/projected/1e6be196-1d2f-4255-a612-151457495a5f-kube-api-access-8lzlp\") pod \"limitador-operator-controller-manager-74495c69f8-79pgz\" (UID: \"1e6be196-1d2f-4255-a612-151457495a5f\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-79pgz" Sep 11 18:23:32.678519 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:32.678306 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbvt8\" (UniqueName: \"kubernetes.io/projected/0059eaea-499d-499c-98d6-9547fc02722d-kube-api-access-cbvt8\") pod \"kuadrant-operator-controller-manager-5769f99d9f-lsktr\" (UID: \"0059eaea-499d-499c-98d6-9547fc02722d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lsktr" Sep 11 18:23:32.678643 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:32.678620 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fce4d870-b756-43ef-b375-3b092192f03c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-5r2mj\" (UID: \"fce4d870-b756-43ef-b375-3b092192f03c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5r2mj" Sep 11 18:23:32.691867 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:32.691835 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scw9x\" (UniqueName: \"kubernetes.io/projected/fce4d870-b756-43ef-b375-3b092192f03c-kube-api-access-scw9x\") pod \"kuadrant-operator-controller-manager-76f8f686bf-5r2mj\" (UID: \"fce4d870-b756-43ef-b375-3b092192f03c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5r2mj" Sep 11 18:23:32.779226 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:32.779197 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lzlp\" (UniqueName: \"kubernetes.io/projected/1e6be196-1d2f-4255-a612-151457495a5f-kube-api-access-8lzlp\") pod \"limitador-operator-controller-manager-74495c69f8-79pgz\" (UID: \"1e6be196-1d2f-4255-a612-151457495a5f\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-79pgz" Sep 11 18:23:32.779331 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:32.779278 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbvt8\" (UniqueName: \"kubernetes.io/projected/0059eaea-499d-499c-98d6-9547fc02722d-kube-api-access-cbvt8\") pod \"kuadrant-operator-controller-manager-5769f99d9f-lsktr\" (UID: \"0059eaea-499d-499c-98d6-9547fc02722d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lsktr" Sep 11 18:23:32.779398 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:32.779382 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0059eaea-499d-499c-98d6-9547fc02722d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-lsktr\" (UID: \"0059eaea-499d-499c-98d6-9547fc02722d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lsktr" Sep 11 18:23:32.779740 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:32.779720 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0059eaea-499d-499c-98d6-9547fc02722d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-lsktr\" (UID: \"0059eaea-499d-499c-98d6-9547fc02722d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lsktr" Sep 11 18:23:32.787243 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:32.787176 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rdn5x" Sep 11 18:23:32.788508 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:32.788488 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbvt8\" (UniqueName: \"kubernetes.io/projected/0059eaea-499d-499c-98d6-9547fc02722d-kube-api-access-cbvt8\") pod \"kuadrant-operator-controller-manager-5769f99d9f-lsktr\" (UID: \"0059eaea-499d-499c-98d6-9547fc02722d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lsktr" Sep 11 18:23:32.788628 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:32.788597 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lzlp\" (UniqueName: \"kubernetes.io/projected/1e6be196-1d2f-4255-a612-151457495a5f-kube-api-access-8lzlp\") pod \"limitador-operator-controller-manager-74495c69f8-79pgz\" (UID: \"1e6be196-1d2f-4255-a612-151457495a5f\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-79pgz" Sep 11 18:23:32.789065 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:23:32.789033 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-rdn5x\" is forbidden: User \"system:node:ip-10-0-138-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-128.ec2.internal' and this object" podUID="012e75f6-47cb-4112-bc94-63c492558cab" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rdn5x" Sep 11 18:23:32.789423 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:32.789407 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qc2xg" Sep 11 18:23:32.790673 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:23:32.790654 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-qc2xg\" is forbidden: User \"system:node:ip-10-0-138-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-128.ec2.internal' and this object" podUID="fe464dd0-84f3-4d43-b74e-f44ca0d4ebf2" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qc2xg" Sep 11 18:23:32.792215 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:23:32.792195 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-rdn5x\" is forbidden: User \"system:node:ip-10-0-138-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-128.ec2.internal' and this object" podUID="012e75f6-47cb-4112-bc94-63c492558cab" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rdn5x" Sep 11 18:23:32.793726 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:23:32.793701 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-qc2xg\" is forbidden: User \"system:node:ip-10-0-138-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-128.ec2.internal' and this object" podUID="fe464dd0-84f3-4d43-b74e-f44ca0d4ebf2" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qc2xg" Sep 11 18:23:32.836819 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:32.836784 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5r2mj" Sep 11 18:23:32.880698 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:32.880661 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xqmvj\" (UniqueName: \"kubernetes.io/projected/012e75f6-47cb-4112-bc94-63c492558cab-kube-api-access-xqmvj\") pod \"012e75f6-47cb-4112-bc94-63c492558cab\" (UID: \"012e75f6-47cb-4112-bc94-63c492558cab\") " Sep 11 18:23:32.880871 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:32.880744 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4v6pg\" (UniqueName: \"kubernetes.io/projected/fe464dd0-84f3-4d43-b74e-f44ca0d4ebf2-kube-api-access-4v6pg\") pod \"fe464dd0-84f3-4d43-b74e-f44ca0d4ebf2\" (UID: \"fe464dd0-84f3-4d43-b74e-f44ca0d4ebf2\") " Sep 11 18:23:32.880871 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:32.880863 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/012e75f6-47cb-4112-bc94-63c492558cab-extensions-socket-volume\") pod \"012e75f6-47cb-4112-bc94-63c492558cab\" (UID: \"012e75f6-47cb-4112-bc94-63c492558cab\") " Sep 11 18:23:32.881368 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:32.881335 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/012e75f6-47cb-4112-bc94-63c492558cab-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "012e75f6-47cb-4112-bc94-63c492558cab" (UID: "012e75f6-47cb-4112-bc94-63c492558cab"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:23:32.882901 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:32.882879 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe464dd0-84f3-4d43-b74e-f44ca0d4ebf2-kube-api-access-4v6pg" (OuterVolumeSpecName: "kube-api-access-4v6pg") pod "fe464dd0-84f3-4d43-b74e-f44ca0d4ebf2" (UID: "fe464dd0-84f3-4d43-b74e-f44ca0d4ebf2"). InnerVolumeSpecName "kube-api-access-4v6pg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:23:32.882986 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:32.882895 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/012e75f6-47cb-4112-bc94-63c492558cab-kube-api-access-xqmvj" (OuterVolumeSpecName: "kube-api-access-xqmvj") pod "012e75f6-47cb-4112-bc94-63c492558cab" (UID: "012e75f6-47cb-4112-bc94-63c492558cab"). InnerVolumeSpecName "kube-api-access-xqmvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:23:32.929192 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:32.929156 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lsktr" Sep 11 18:23:32.934709 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:32.934666 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-79pgz" Sep 11 18:23:32.982167 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:32.982130 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v6pg\" (UniqueName: \"kubernetes.io/projected/fe464dd0-84f3-4d43-b74e-f44ca0d4ebf2-kube-api-access-4v6pg\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:23:32.982167 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:32.982163 2561 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/012e75f6-47cb-4112-bc94-63c492558cab-extensions-socket-volume\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:23:32.982167 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:32.982175 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqmvj\" (UniqueName: \"kubernetes.io/projected/012e75f6-47cb-4112-bc94-63c492558cab-kube-api-access-xqmvj\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:23:33.116295 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:33.116270 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5r2mj"] Sep 11 18:23:33.118321 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:23:33.118293 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfce4d870_b756_43ef_b375_3b092192f03c.slice/crio-1e12f5d5080318555aea6fac18038f683e6c2555d57dbd6bf7ee88b7d833bfda WatchSource:0}: Error finding container 1e12f5d5080318555aea6fac18038f683e6c2555d57dbd6bf7ee88b7d833bfda: Status 404 returned error can't find the container with id 1e12f5d5080318555aea6fac18038f683e6c2555d57dbd6bf7ee88b7d833bfda Sep 11 18:23:33.139287 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:33.139260 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="012e75f6-47cb-4112-bc94-63c492558cab" path="/var/lib/kubelet/pods/012e75f6-47cb-4112-bc94-63c492558cab/volumes" Sep 11 18:23:33.139722 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:33.139707 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe464dd0-84f3-4d43-b74e-f44ca0d4ebf2" path="/var/lib/kubelet/pods/fe464dd0-84f3-4d43-b74e-f44ca0d4ebf2/volumes" Sep 11 18:23:33.223855 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:33.223825 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-79pgz"] Sep 11 18:23:33.225828 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:23:33.225798 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e6be196_1d2f_4255_a612_151457495a5f.slice/crio-62b9c1b55eea66f0b1fc0502e0f4f02f33b60fecabce5e123274c5d7912832f5 WatchSource:0}: Error finding container 62b9c1b55eea66f0b1fc0502e0f4f02f33b60fecabce5e123274c5d7912832f5: Status 404 returned error can't find the container with id 62b9c1b55eea66f0b1fc0502e0f4f02f33b60fecabce5e123274c5d7912832f5 Sep 11 18:23:33.227942 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:33.227913 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lsktr"] Sep 11 18:23:33.228926 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:23:33.228900 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0059eaea_499d_499c_98d6_9547fc02722d.slice/crio-e5fbcf9437aa5c3fbe774a50507db4b4370bfc773475e5612609582bc67c70fa WatchSource:0}: Error finding container e5fbcf9437aa5c3fbe774a50507db4b4370bfc773475e5612609582bc67c70fa: Status 404 returned error can't find the container with id e5fbcf9437aa5c3fbe774a50507db4b4370bfc773475e5612609582bc67c70fa Sep 11 18:23:33.451486 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:33.451447 2561 generic.go:356] "Generic (PLEG): container finished" podID="012e75f6-47cb-4112-bc94-63c492558cab" containerID="68c83127e3338d0f70d9811f65696371d20f8a0e6621c6ab07e62e98045caf6d" exitCode=0 Sep 11 18:23:33.451639 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:33.451500 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rdn5x" Sep 11 18:23:33.451639 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:33.451550 2561 scope.go:117] "RemoveContainer" containerID="68c83127e3338d0f70d9811f65696371d20f8a0e6621c6ab07e62e98045caf6d" Sep 11 18:23:33.453608 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:33.453579 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-79pgz" event={"ID":"1e6be196-1d2f-4255-a612-151457495a5f","Type":"ContainerStarted","Data":"f16fb4d11b7fb6addce8a6a2438df9703a95569095a1ba9c0420c4a2cad5d29e"} Sep 11 18:23:33.453740 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:33.453654 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-79pgz" Sep 11 18:23:33.453740 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:33.453682 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-79pgz" event={"ID":"1e6be196-1d2f-4255-a612-151457495a5f","Type":"ContainerStarted","Data":"62b9c1b55eea66f0b1fc0502e0f4f02f33b60fecabce5e123274c5d7912832f5"} Sep 11 18:23:33.455409 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:33.455387 2561 generic.go:356] "Generic (PLEG): container finished" podID="fe464dd0-84f3-4d43-b74e-f44ca0d4ebf2" containerID="1f6d24e3ed77579184cacee0f0bf65ed83d3353c13fa5fc9dff659c2684ae80e" exitCode=0 Sep 11 18:23:33.455718 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:33.455443 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qc2xg" Sep 11 18:23:33.457555 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:33.457529 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5r2mj" event={"ID":"fce4d870-b756-43ef-b375-3b092192f03c","Type":"ContainerStarted","Data":"83aa2b2cf91c125f1e3de7b56c4b2593add7b68608ada25cea66cec399b22cdf"} Sep 11 18:23:33.457672 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:33.457586 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5r2mj" Sep 11 18:23:33.457672 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:33.457613 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5r2mj" event={"ID":"fce4d870-b756-43ef-b375-3b092192f03c","Type":"ContainerStarted","Data":"1e12f5d5080318555aea6fac18038f683e6c2555d57dbd6bf7ee88b7d833bfda"} Sep 11 18:23:33.459561 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:33.459541 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6zfkv" Sep 11 18:23:33.459662 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:33.459568 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lsktr" event={"ID":"0059eaea-499d-499c-98d6-9547fc02722d","Type":"ContainerStarted","Data":"5f3653ee21cd4951b8d64ae6e2780bd1b6d3209aa53a3fc3f2a62995319f9f39"} Sep 11 18:23:33.459662 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:33.459613 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lsktr" Sep 11 18:23:33.459662 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:33.459632 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lsktr" event={"ID":"0059eaea-499d-499c-98d6-9547fc02722d","Type":"ContainerStarted","Data":"e5fbcf9437aa5c3fbe774a50507db4b4370bfc773475e5612609582bc67c70fa"} Sep 11 18:23:33.462823 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:33.462639 2561 scope.go:117] "RemoveContainer" containerID="68c83127e3338d0f70d9811f65696371d20f8a0e6621c6ab07e62e98045caf6d" Sep 11 18:23:33.462912 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:23:33.462897 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68c83127e3338d0f70d9811f65696371d20f8a0e6621c6ab07e62e98045caf6d\": container with ID starting with 68c83127e3338d0f70d9811f65696371d20f8a0e6621c6ab07e62e98045caf6d not found: ID does not exist" containerID="68c83127e3338d0f70d9811f65696371d20f8a0e6621c6ab07e62e98045caf6d" Sep 11 18:23:33.462955 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:33.462919 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68c83127e3338d0f70d9811f65696371d20f8a0e6621c6ab07e62e98045caf6d"} err="failed to get container status \"68c83127e3338d0f70d9811f65696371d20f8a0e6621c6ab07e62e98045caf6d\": rpc error: code = NotFound desc = could not find container \"68c83127e3338d0f70d9811f65696371d20f8a0e6621c6ab07e62e98045caf6d\": container with ID starting with 68c83127e3338d0f70d9811f65696371d20f8a0e6621c6ab07e62e98045caf6d not found: ID does not exist" Sep 11 18:23:33.462955 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:33.462933 2561 scope.go:117] "RemoveContainer" containerID="1f6d24e3ed77579184cacee0f0bf65ed83d3353c13fa5fc9dff659c2684ae80e" Sep 11 18:23:33.467322 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:33.467307 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6zfkv" Sep 11 18:23:33.472632 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:33.472591 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-79pgz" podStartSLOduration=1.4725779110000001 podStartE2EDuration="1.472577911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:23:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:23:33.471704272 +0000 UTC m=+494.968722027" watchObservedRunningTime="2026-09-11 18:23:33.472577911 +0000 UTC m=+494.969595664" Sep 11 18:23:33.493691 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:33.493664 2561 scope.go:117] "RemoveContainer" containerID="1f6d24e3ed77579184cacee0f0bf65ed83d3353c13fa5fc9dff659c2684ae80e" Sep 11 18:23:33.494204 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:23:33.494174 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f6d24e3ed77579184cacee0f0bf65ed83d3353c13fa5fc9dff659c2684ae80e\": container with ID starting with 1f6d24e3ed77579184cacee0f0bf65ed83d3353c13fa5fc9dff659c2684ae80e not found: ID does not exist" containerID="1f6d24e3ed77579184cacee0f0bf65ed83d3353c13fa5fc9dff659c2684ae80e" Sep 11 18:23:33.494327 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:33.494213 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f6d24e3ed77579184cacee0f0bf65ed83d3353c13fa5fc9dff659c2684ae80e"} err="failed to get container status \"1f6d24e3ed77579184cacee0f0bf65ed83d3353c13fa5fc9dff659c2684ae80e\": rpc error: code = NotFound desc = could not find container \"1f6d24e3ed77579184cacee0f0bf65ed83d3353c13fa5fc9dff659c2684ae80e\": container with ID starting with 1f6d24e3ed77579184cacee0f0bf65ed83d3353c13fa5fc9dff659c2684ae80e not found: ID does not exist" Sep 11 18:23:33.512409 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:33.512344 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5r2mj" podStartSLOduration=1.512325015 podStartE2EDuration="1.512325015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:23:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:23:33.494031473 +0000 UTC m=+494.991049255" watchObservedRunningTime="2026-09-11 18:23:33.512325015 +0000 UTC m=+495.009342770" Sep 11 18:23:33.513836 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:23:33.513800 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-6zfkv\" is forbidden: User \"system:node:ip-10-0-138-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-128.ec2.internal' and this object" podUID="b487f8fd-50ca-453d-8e2a-3203520aa888" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6zfkv" Sep 11 18:23:33.513982 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:33.513841 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lsktr" podStartSLOduration=1.513828373 podStartE2EDuration="1.513828373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:23:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:23:33.511752431 +0000 UTC m=+495.008770186" watchObservedRunningTime="2026-09-11 18:23:33.513828373 +0000 UTC m=+495.010846128" Sep 11 18:23:33.586939 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:33.586893 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b487f8fd-50ca-453d-8e2a-3203520aa888-extensions-socket-volume\") pod \"b487f8fd-50ca-453d-8e2a-3203520aa888\" (UID: \"b487f8fd-50ca-453d-8e2a-3203520aa888\") " Sep 11 18:23:33.587132 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:33.586961 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b4h6p\" (UniqueName: \"kubernetes.io/projected/b487f8fd-50ca-453d-8e2a-3203520aa888-kube-api-access-b4h6p\") pod \"b487f8fd-50ca-453d-8e2a-3203520aa888\" (UID: \"b487f8fd-50ca-453d-8e2a-3203520aa888\") " Sep 11 18:23:33.587225 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:33.587194 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b487f8fd-50ca-453d-8e2a-3203520aa888-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b487f8fd-50ca-453d-8e2a-3203520aa888" (UID: "b487f8fd-50ca-453d-8e2a-3203520aa888"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:23:33.587344 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:33.587326 2561 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b487f8fd-50ca-453d-8e2a-3203520aa888-extensions-socket-volume\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:23:33.589536 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:33.589506 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b487f8fd-50ca-453d-8e2a-3203520aa888-kube-api-access-b4h6p" (OuterVolumeSpecName: "kube-api-access-b4h6p") pod "b487f8fd-50ca-453d-8e2a-3203520aa888" (UID: "b487f8fd-50ca-453d-8e2a-3203520aa888"). InnerVolumeSpecName "kube-api-access-b4h6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:23:33.688733 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:33.688694 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4h6p\" (UniqueName: \"kubernetes.io/projected/b487f8fd-50ca-453d-8e2a-3203520aa888-kube-api-access-b4h6p\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:23:34.467979 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:34.467950 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6zfkv" Sep 11 18:23:34.470078 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:23:34.470028 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-6zfkv\" is forbidden: User \"system:node:ip-10-0-138-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-128.ec2.internal' and this object" podUID="b487f8fd-50ca-453d-8e2a-3203520aa888" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6zfkv" Sep 11 18:23:34.475742 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:23:34.475701 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-6zfkv\" is forbidden: User \"system:node:ip-10-0-138-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-128.ec2.internal' and this object" podUID="b487f8fd-50ca-453d-8e2a-3203520aa888" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6zfkv" Sep 11 18:23:35.139848 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:35.139813 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b487f8fd-50ca-453d-8e2a-3203520aa888" path="/var/lib/kubelet/pods/b487f8fd-50ca-453d-8e2a-3203520aa888/volumes" Sep 11 18:23:44.469744 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:44.469712 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-79pgz" Sep 11 18:23:44.470298 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:44.469800 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lsktr" Sep 11 18:23:44.470298 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:44.469980 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5r2mj" Sep 11 18:23:44.521805 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:44.521772 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5r2mj"] Sep 11 18:23:44.526936 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:44.526902 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5r2mj" podUID="fce4d870-b756-43ef-b375-3b092192f03c" containerName="manager" containerID="cri-o://83aa2b2cf91c125f1e3de7b56c4b2593add7b68608ada25cea66cec399b22cdf" gracePeriod=10 Sep 11 18:23:44.772733 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:44.772710 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5r2mj" Sep 11 18:23:44.883943 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:44.883903 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fce4d870-b756-43ef-b375-3b092192f03c-extensions-socket-volume\") pod \"fce4d870-b756-43ef-b375-3b092192f03c\" (UID: \"fce4d870-b756-43ef-b375-3b092192f03c\") " Sep 11 18:23:44.884179 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:44.884042 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-scw9x\" (UniqueName: \"kubernetes.io/projected/fce4d870-b756-43ef-b375-3b092192f03c-kube-api-access-scw9x\") pod \"fce4d870-b756-43ef-b375-3b092192f03c\" (UID: \"fce4d870-b756-43ef-b375-3b092192f03c\") " Sep 11 18:23:44.884329 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:44.884291 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fce4d870-b756-43ef-b375-3b092192f03c-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "fce4d870-b756-43ef-b375-3b092192f03c" (UID: "fce4d870-b756-43ef-b375-3b092192f03c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:23:44.886077 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:44.886040 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fce4d870-b756-43ef-b375-3b092192f03c-kube-api-access-scw9x" (OuterVolumeSpecName: "kube-api-access-scw9x") pod "fce4d870-b756-43ef-b375-3b092192f03c" (UID: "fce4d870-b756-43ef-b375-3b092192f03c"). InnerVolumeSpecName "kube-api-access-scw9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:23:44.984679 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:44.984642 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scw9x\" (UniqueName: \"kubernetes.io/projected/fce4d870-b756-43ef-b375-3b092192f03c-kube-api-access-scw9x\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:23:44.984679 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:44.984672 2561 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fce4d870-b756-43ef-b375-3b092192f03c-extensions-socket-volume\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:23:45.533414 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:45.533378 2561 generic.go:356] "Generic (PLEG): container finished" podID="fce4d870-b756-43ef-b375-3b092192f03c" containerID="83aa2b2cf91c125f1e3de7b56c4b2593add7b68608ada25cea66cec399b22cdf" exitCode=0 Sep 11 18:23:45.533860 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:45.533437 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5r2mj" Sep 11 18:23:45.533860 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:45.533458 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5r2mj" event={"ID":"fce4d870-b756-43ef-b375-3b092192f03c","Type":"ContainerDied","Data":"83aa2b2cf91c125f1e3de7b56c4b2593add7b68608ada25cea66cec399b22cdf"} Sep 11 18:23:45.533860 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:45.533509 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5r2mj" event={"ID":"fce4d870-b756-43ef-b375-3b092192f03c","Type":"ContainerDied","Data":"1e12f5d5080318555aea6fac18038f683e6c2555d57dbd6bf7ee88b7d833bfda"} Sep 11 18:23:45.533860 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:45.533530 2561 scope.go:117] "RemoveContainer" containerID="83aa2b2cf91c125f1e3de7b56c4b2593add7b68608ada25cea66cec399b22cdf" Sep 11 18:23:45.541971 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:45.541954 2561 scope.go:117] "RemoveContainer" containerID="83aa2b2cf91c125f1e3de7b56c4b2593add7b68608ada25cea66cec399b22cdf" Sep 11 18:23:45.542253 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:23:45.542231 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83aa2b2cf91c125f1e3de7b56c4b2593add7b68608ada25cea66cec399b22cdf\": container with ID starting with 83aa2b2cf91c125f1e3de7b56c4b2593add7b68608ada25cea66cec399b22cdf not found: ID does not exist" containerID="83aa2b2cf91c125f1e3de7b56c4b2593add7b68608ada25cea66cec399b22cdf" Sep 11 18:23:45.542333 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:45.542261 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83aa2b2cf91c125f1e3de7b56c4b2593add7b68608ada25cea66cec399b22cdf"} err="failed to get container status \"83aa2b2cf91c125f1e3de7b56c4b2593add7b68608ada25cea66cec399b22cdf\": rpc error: code = NotFound desc = could not find container \"83aa2b2cf91c125f1e3de7b56c4b2593add7b68608ada25cea66cec399b22cdf\": container with ID starting with 83aa2b2cf91c125f1e3de7b56c4b2593add7b68608ada25cea66cec399b22cdf not found: ID does not exist" Sep 11 18:23:45.560688 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:45.560663 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5r2mj"] Sep 11 18:23:45.565949 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:45.565925 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5r2mj"] Sep 11 18:23:47.139013 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:23:47.138977 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fce4d870-b756-43ef-b375-3b092192f03c" path="/var/lib/kubelet/pods/fce4d870-b756-43ef-b375-3b092192f03c/volumes" Sep 11 18:24:00.825617 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:00.825581 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cbvsl"] Sep 11 18:24:00.826101 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:00.826011 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fce4d870-b756-43ef-b375-3b092192f03c" containerName="manager" Sep 11 18:24:00.826101 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:00.826028 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="fce4d870-b756-43ef-b375-3b092192f03c" containerName="manager" Sep 11 18:24:00.826220 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:00.826108 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="fce4d870-b756-43ef-b375-3b092192f03c" containerName="manager" Sep 11 18:24:00.828746 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:00.828721 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cbvsl" Sep 11 18:24:00.830989 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:00.830964 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-d6qxw\"" Sep 11 18:24:00.842470 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:00.842442 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cbvsl"] Sep 11 18:24:00.929155 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:00.929113 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09816140-ebbc-4f2d-9e1a-a11e86d773a4-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-cbvsl\" (UID: \"09816140-ebbc-4f2d-9e1a-a11e86d773a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cbvsl" Sep 11 18:24:00.929155 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:00.929152 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09816140-ebbc-4f2d-9e1a-a11e86d773a4-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-cbvsl\" (UID: \"09816140-ebbc-4f2d-9e1a-a11e86d773a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cbvsl" Sep 11 18:24:00.929382 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:00.929175 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qpt2\" (UniqueName: \"kubernetes.io/projected/09816140-ebbc-4f2d-9e1a-a11e86d773a4-kube-api-access-5qpt2\") pod \"maas-default-gateway-openshift-default-78bb78ff78-cbvsl\" (UID: \"09816140-ebbc-4f2d-9e1a-a11e86d773a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cbvsl" Sep 11 18:24:00.929382 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:00.929206 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09816140-ebbc-4f2d-9e1a-a11e86d773a4-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-cbvsl\" (UID: \"09816140-ebbc-4f2d-9e1a-a11e86d773a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cbvsl" Sep 11 18:24:00.929382 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:00.929222 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09816140-ebbc-4f2d-9e1a-a11e86d773a4-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-cbvsl\" (UID: \"09816140-ebbc-4f2d-9e1a-a11e86d773a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cbvsl" Sep 11 18:24:00.929382 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:00.929276 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09816140-ebbc-4f2d-9e1a-a11e86d773a4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-cbvsl\" (UID: \"09816140-ebbc-4f2d-9e1a-a11e86d773a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cbvsl" Sep 11 18:24:00.929382 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:00.929307 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09816140-ebbc-4f2d-9e1a-a11e86d773a4-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-cbvsl\" (UID: \"09816140-ebbc-4f2d-9e1a-a11e86d773a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cbvsl" Sep 11 18:24:00.929382 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:00.929345 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09816140-ebbc-4f2d-9e1a-a11e86d773a4-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-cbvsl\" (UID: \"09816140-ebbc-4f2d-9e1a-a11e86d773a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cbvsl" Sep 11 18:24:00.929611 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:00.929406 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09816140-ebbc-4f2d-9e1a-a11e86d773a4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-cbvsl\" (UID: \"09816140-ebbc-4f2d-9e1a-a11e86d773a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cbvsl" Sep 11 18:24:01.030410 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:01.030375 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09816140-ebbc-4f2d-9e1a-a11e86d773a4-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-cbvsl\" (UID: \"09816140-ebbc-4f2d-9e1a-a11e86d773a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cbvsl" Sep 11 18:24:01.030585 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:01.030420 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09816140-ebbc-4f2d-9e1a-a11e86d773a4-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-cbvsl\" (UID: \"09816140-ebbc-4f2d-9e1a-a11e86d773a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cbvsl" Sep 11 18:24:01.030585 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:01.030459 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09816140-ebbc-4f2d-9e1a-a11e86d773a4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-cbvsl\" (UID: \"09816140-ebbc-4f2d-9e1a-a11e86d773a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cbvsl" Sep 11 18:24:01.030585 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:01.030483 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09816140-ebbc-4f2d-9e1a-a11e86d773a4-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-cbvsl\" (UID: \"09816140-ebbc-4f2d-9e1a-a11e86d773a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cbvsl" Sep 11 18:24:01.030585 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:01.030497 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09816140-ebbc-4f2d-9e1a-a11e86d773a4-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-cbvsl\" (UID: \"09816140-ebbc-4f2d-9e1a-a11e86d773a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cbvsl" Sep 11 18:24:01.030585 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:01.030516 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qpt2\" (UniqueName: \"kubernetes.io/projected/09816140-ebbc-4f2d-9e1a-a11e86d773a4-kube-api-access-5qpt2\") pod \"maas-default-gateway-openshift-default-78bb78ff78-cbvsl\" (UID: \"09816140-ebbc-4f2d-9e1a-a11e86d773a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cbvsl" Sep 11 18:24:01.030585 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:01.030554 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09816140-ebbc-4f2d-9e1a-a11e86d773a4-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-cbvsl\" (UID: \"09816140-ebbc-4f2d-9e1a-a11e86d773a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cbvsl" Sep 11 18:24:01.030585 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:01.030578 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09816140-ebbc-4f2d-9e1a-a11e86d773a4-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-cbvsl\" (UID: \"09816140-ebbc-4f2d-9e1a-a11e86d773a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cbvsl" Sep 11 18:24:01.030952 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:01.030618 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09816140-ebbc-4f2d-9e1a-a11e86d773a4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-cbvsl\" (UID: \"09816140-ebbc-4f2d-9e1a-a11e86d773a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cbvsl" Sep 11 18:24:01.031012 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:01.030983 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09816140-ebbc-4f2d-9e1a-a11e86d773a4-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-cbvsl\" (UID: \"09816140-ebbc-4f2d-9e1a-a11e86d773a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cbvsl" Sep 11 18:24:01.031012 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:01.031000 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09816140-ebbc-4f2d-9e1a-a11e86d773a4-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-cbvsl\" (UID: \"09816140-ebbc-4f2d-9e1a-a11e86d773a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cbvsl" Sep 11 18:24:01.031138 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:01.031087 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09816140-ebbc-4f2d-9e1a-a11e86d773a4-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-cbvsl\" (UID: \"09816140-ebbc-4f2d-9e1a-a11e86d773a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cbvsl" Sep 11 18:24:01.031238 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:01.031217 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09816140-ebbc-4f2d-9e1a-a11e86d773a4-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-cbvsl\" (UID: \"09816140-ebbc-4f2d-9e1a-a11e86d773a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cbvsl" Sep 11 18:24:01.031415 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:01.031397 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09816140-ebbc-4f2d-9e1a-a11e86d773a4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-cbvsl\" (UID: \"09816140-ebbc-4f2d-9e1a-a11e86d773a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cbvsl" Sep 11 18:24:01.033262 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:01.033235 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09816140-ebbc-4f2d-9e1a-a11e86d773a4-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-cbvsl\" (UID: \"09816140-ebbc-4f2d-9e1a-a11e86d773a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cbvsl" Sep 11 18:24:01.033447 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:01.033430 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09816140-ebbc-4f2d-9e1a-a11e86d773a4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-cbvsl\" (UID: \"09816140-ebbc-4f2d-9e1a-a11e86d773a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cbvsl" Sep 11 18:24:01.038929 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:01.038897 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qpt2\" (UniqueName: \"kubernetes.io/projected/09816140-ebbc-4f2d-9e1a-a11e86d773a4-kube-api-access-5qpt2\") pod \"maas-default-gateway-openshift-default-78bb78ff78-cbvsl\" (UID: \"09816140-ebbc-4f2d-9e1a-a11e86d773a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cbvsl" Sep 11 18:24:01.038929 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:01.038919 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09816140-ebbc-4f2d-9e1a-a11e86d773a4-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-cbvsl\" (UID: \"09816140-ebbc-4f2d-9e1a-a11e86d773a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cbvsl" Sep 11 18:24:01.140992 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:01.140907 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cbvsl" Sep 11 18:24:01.417439 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:01.417406 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cbvsl"] Sep 11 18:24:01.418313 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:24:01.418285 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09816140_ebbc_4f2d_9e1a_a11e86d773a4.slice/crio-56f382847ad94a43a1c1662126eaf3f342f0c9ff31fe4191dc4f658caf3ef13b WatchSource:0}: Error finding container 56f382847ad94a43a1c1662126eaf3f342f0c9ff31fe4191dc4f658caf3ef13b: Status 404 returned error can't find the container with id 56f382847ad94a43a1c1662126eaf3f342f0c9ff31fe4191dc4f658caf3ef13b Sep 11 18:24:01.420534 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:01.420496 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:24:01.420632 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:01.420568 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:24:01.420632 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:01.420600 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:24:01.625554 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:01.625521 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cbvsl" event={"ID":"09816140-ebbc-4f2d-9e1a-a11e86d773a4","Type":"ContainerStarted","Data":"7035135a4da19409fb74bb8b6e3c17a77f63c3e31a82d1e5df51fa919ac2e6f7"} Sep 11 18:24:01.625760 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:01.625583 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cbvsl" event={"ID":"09816140-ebbc-4f2d-9e1a-a11e86d773a4","Type":"ContainerStarted","Data":"56f382847ad94a43a1c1662126eaf3f342f0c9ff31fe4191dc4f658caf3ef13b"} Sep 11 18:24:01.645210 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:01.645155 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cbvsl" podStartSLOduration=1.645132704 podStartE2EDuration="1.645132704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:24:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:24:01.644035261 +0000 UTC m=+523.141053016" watchObservedRunningTime="2026-09-11 18:24:01.645132704 +0000 UTC m=+523.142150460" Sep 11 18:24:02.142003 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:02.141963 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cbvsl" Sep 11 18:24:02.147131 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:02.147106 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cbvsl" Sep 11 18:24:02.632010 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:02.631976 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cbvsl" Sep 11 18:24:02.633030 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:02.633009 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cbvsl" Sep 11 18:24:03.509820 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:03.509782 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-p5929"] Sep 11 18:24:03.512783 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:03.512759 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-p5929" Sep 11 18:24:03.515262 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:03.515236 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-vzjbs\"" Sep 11 18:24:03.523533 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:03.523505 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-p5929"] Sep 11 18:24:03.655526 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:03.655495 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbmhv\" (UniqueName: \"kubernetes.io/projected/90b4dd1c-4bcc-4ae2-945e-3cdd5946bf1b-kube-api-access-qbmhv\") pod \"authorino-7bcfd8f9cd-p5929\" (UID: \"90b4dd1c-4bcc-4ae2-945e-3cdd5946bf1b\") " pod="kuadrant-system/authorino-7bcfd8f9cd-p5929" Sep 11 18:24:03.668715 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:03.668668 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-qnmrd"] Sep 11 18:24:03.671714 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:03.671690 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-qnmrd" Sep 11 18:24:03.681672 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:03.681646 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-qnmrd"] Sep 11 18:24:03.757109 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:03.756986 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbmhv\" (UniqueName: \"kubernetes.io/projected/90b4dd1c-4bcc-4ae2-945e-3cdd5946bf1b-kube-api-access-qbmhv\") pod \"authorino-7bcfd8f9cd-p5929\" (UID: \"90b4dd1c-4bcc-4ae2-945e-3cdd5946bf1b\") " pod="kuadrant-system/authorino-7bcfd8f9cd-p5929" Sep 11 18:24:03.757328 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:03.757303 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9dzk\" (UniqueName: \"kubernetes.io/projected/a7fb7861-dc94-40d9-ae45-047104357cfb-kube-api-access-s9dzk\") pod \"authorino-59c974bd48-qnmrd\" (UID: \"a7fb7861-dc94-40d9-ae45-047104357cfb\") " pod="kuadrant-system/authorino-59c974bd48-qnmrd" Sep 11 18:24:03.765700 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:03.765636 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbmhv\" (UniqueName: \"kubernetes.io/projected/90b4dd1c-4bcc-4ae2-945e-3cdd5946bf1b-kube-api-access-qbmhv\") pod \"authorino-7bcfd8f9cd-p5929\" (UID: \"90b4dd1c-4bcc-4ae2-945e-3cdd5946bf1b\") " pod="kuadrant-system/authorino-7bcfd8f9cd-p5929" Sep 11 18:24:03.823717 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:03.823689 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-p5929" Sep 11 18:24:03.858610 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:03.858580 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9dzk\" (UniqueName: \"kubernetes.io/projected/a7fb7861-dc94-40d9-ae45-047104357cfb-kube-api-access-s9dzk\") pod \"authorino-59c974bd48-qnmrd\" (UID: \"a7fb7861-dc94-40d9-ae45-047104357cfb\") " pod="kuadrant-system/authorino-59c974bd48-qnmrd" Sep 11 18:24:03.867845 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:03.867819 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9dzk\" (UniqueName: \"kubernetes.io/projected/a7fb7861-dc94-40d9-ae45-047104357cfb-kube-api-access-s9dzk\") pod \"authorino-59c974bd48-qnmrd\" (UID: \"a7fb7861-dc94-40d9-ae45-047104357cfb\") " pod="kuadrant-system/authorino-59c974bd48-qnmrd" Sep 11 18:24:03.980741 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:03.980708 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-qnmrd" Sep 11 18:24:04.094532 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:04.094507 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-p5929"] Sep 11 18:24:04.096500 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:24:04.096474 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod90b4dd1c_4bcc_4ae2_945e_3cdd5946bf1b.slice/crio-489f79c9394c8df5cca674747938c9beaad497444052f1a0724dbe3a136d7090 WatchSource:0}: Error finding container 489f79c9394c8df5cca674747938c9beaad497444052f1a0724dbe3a136d7090: Status 404 returned error can't find the container with id 489f79c9394c8df5cca674747938c9beaad497444052f1a0724dbe3a136d7090 Sep 11 18:24:04.252902 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:04.252875 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-qnmrd"] Sep 11 18:24:04.254516 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:24:04.254483 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda7fb7861_dc94_40d9_ae45_047104357cfb.slice/crio-a23cbbe18013c276dd7f16aac35daa0fd27bed3d15fe7b276ea925c34f94a3a0 WatchSource:0}: Error finding container a23cbbe18013c276dd7f16aac35daa0fd27bed3d15fe7b276ea925c34f94a3a0: Status 404 returned error can't find the container with id a23cbbe18013c276dd7f16aac35daa0fd27bed3d15fe7b276ea925c34f94a3a0 Sep 11 18:24:04.644677 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:04.644639 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-p5929" event={"ID":"90b4dd1c-4bcc-4ae2-945e-3cdd5946bf1b","Type":"ContainerStarted","Data":"489f79c9394c8df5cca674747938c9beaad497444052f1a0724dbe3a136d7090"} Sep 11 18:24:04.645880 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:04.645853 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-qnmrd" event={"ID":"a7fb7861-dc94-40d9-ae45-047104357cfb","Type":"ContainerStarted","Data":"a23cbbe18013c276dd7f16aac35daa0fd27bed3d15fe7b276ea925c34f94a3a0"} Sep 11 18:24:06.659544 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:06.659505 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-p5929" event={"ID":"90b4dd1c-4bcc-4ae2-945e-3cdd5946bf1b","Type":"ContainerStarted","Data":"88dc2e0980b1bb23a1ff60db24fa727f9d2145d4a15e419fa68cbf897d4387e7"} Sep 11 18:24:06.661103 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:06.661076 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-qnmrd" event={"ID":"a7fb7861-dc94-40d9-ae45-047104357cfb","Type":"ContainerStarted","Data":"97ac26558023f08c723bcdd50691bd18229b6e8b9d05d7559ae02ba506d73346"} Sep 11 18:24:06.674006 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:06.673954 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-p5929" podStartSLOduration=2.085067198 podStartE2EDuration="3.673935211s" totalImagesPullingTime="1.588868013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:24:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:24:06.673612741 +0000 UTC m=+528.170630494" watchObservedRunningTime="2026-09-11 18:24:06.673935211 +0000 UTC m=+528.170952969" Sep 11 18:24:06.688578 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:06.688525 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-qnmrd" podStartSLOduration=2.253555319 podStartE2EDuration="3.688496884s" totalImagesPullingTime="1.434941565s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:24:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:24:06.687893687 +0000 UTC m=+528.184911443" watchObservedRunningTime="2026-09-11 18:24:06.688496884 +0000 UTC m=+528.185514638" Sep 11 18:24:06.717551 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:06.717307 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-p5929"] Sep 11 18:24:08.672398 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:08.672360 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-p5929" podUID="90b4dd1c-4bcc-4ae2-945e-3cdd5946bf1b" containerName="authorino" containerID="cri-o://88dc2e0980b1bb23a1ff60db24fa727f9d2145d4a15e419fa68cbf897d4387e7" gracePeriod=30 Sep 11 18:24:08.916117 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:08.916091 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-p5929" Sep 11 18:24:09.004163 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:09.004039 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qbmhv\" (UniqueName: \"kubernetes.io/projected/90b4dd1c-4bcc-4ae2-945e-3cdd5946bf1b-kube-api-access-qbmhv\") pod \"90b4dd1c-4bcc-4ae2-945e-3cdd5946bf1b\" (UID: \"90b4dd1c-4bcc-4ae2-945e-3cdd5946bf1b\") " Sep 11 18:24:09.006176 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:09.006143 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90b4dd1c-4bcc-4ae2-945e-3cdd5946bf1b-kube-api-access-qbmhv" (OuterVolumeSpecName: "kube-api-access-qbmhv") pod "90b4dd1c-4bcc-4ae2-945e-3cdd5946bf1b" (UID: "90b4dd1c-4bcc-4ae2-945e-3cdd5946bf1b"). InnerVolumeSpecName "kube-api-access-qbmhv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:24:09.104976 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:09.104940 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbmhv\" (UniqueName: \"kubernetes.io/projected/90b4dd1c-4bcc-4ae2-945e-3cdd5946bf1b-kube-api-access-qbmhv\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:24:09.679674 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:09.679633 2561 generic.go:356] "Generic (PLEG): container finished" podID="90b4dd1c-4bcc-4ae2-945e-3cdd5946bf1b" containerID="88dc2e0980b1bb23a1ff60db24fa727f9d2145d4a15e419fa68cbf897d4387e7" exitCode=0 Sep 11 18:24:09.680124 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:09.679696 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-p5929" Sep 11 18:24:09.680124 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:09.679713 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-p5929" event={"ID":"90b4dd1c-4bcc-4ae2-945e-3cdd5946bf1b","Type":"ContainerDied","Data":"88dc2e0980b1bb23a1ff60db24fa727f9d2145d4a15e419fa68cbf897d4387e7"} Sep 11 18:24:09.680124 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:09.679765 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-p5929" event={"ID":"90b4dd1c-4bcc-4ae2-945e-3cdd5946bf1b","Type":"ContainerDied","Data":"489f79c9394c8df5cca674747938c9beaad497444052f1a0724dbe3a136d7090"} Sep 11 18:24:09.680124 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:09.679783 2561 scope.go:117] "RemoveContainer" containerID="88dc2e0980b1bb23a1ff60db24fa727f9d2145d4a15e419fa68cbf897d4387e7" Sep 11 18:24:09.688691 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:09.688673 2561 scope.go:117] "RemoveContainer" containerID="88dc2e0980b1bb23a1ff60db24fa727f9d2145d4a15e419fa68cbf897d4387e7" Sep 11 18:24:09.688954 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:24:09.688937 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88dc2e0980b1bb23a1ff60db24fa727f9d2145d4a15e419fa68cbf897d4387e7\": container with ID starting with 88dc2e0980b1bb23a1ff60db24fa727f9d2145d4a15e419fa68cbf897d4387e7 not found: ID does not exist" containerID="88dc2e0980b1bb23a1ff60db24fa727f9d2145d4a15e419fa68cbf897d4387e7" Sep 11 18:24:09.689001 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:09.688962 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88dc2e0980b1bb23a1ff60db24fa727f9d2145d4a15e419fa68cbf897d4387e7"} err="failed to get container status \"88dc2e0980b1bb23a1ff60db24fa727f9d2145d4a15e419fa68cbf897d4387e7\": rpc error: code = NotFound desc = could not find container \"88dc2e0980b1bb23a1ff60db24fa727f9d2145d4a15e419fa68cbf897d4387e7\": container with ID starting with 88dc2e0980b1bb23a1ff60db24fa727f9d2145d4a15e419fa68cbf897d4387e7 not found: ID does not exist" Sep 11 18:24:09.698502 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:09.698478 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-p5929"] Sep 11 18:24:09.702456 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:09.702422 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-p5929"] Sep 11 18:24:11.138730 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:11.138694 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90b4dd1c-4bcc-4ae2-945e-3cdd5946bf1b" path="/var/lib/kubelet/pods/90b4dd1c-4bcc-4ae2-945e-3cdd5946bf1b/volumes" Sep 11 18:24:32.083041 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:32.083003 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-kjc2z"] Sep 11 18:24:32.083517 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:32.083472 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90b4dd1c-4bcc-4ae2-945e-3cdd5946bf1b" containerName="authorino" Sep 11 18:24:32.083517 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:32.083490 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="90b4dd1c-4bcc-4ae2-945e-3cdd5946bf1b" containerName="authorino" Sep 11 18:24:32.083599 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:32.083546 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="90b4dd1c-4bcc-4ae2-945e-3cdd5946bf1b" containerName="authorino" Sep 11 18:24:32.086555 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:32.086532 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-kjc2z" Sep 11 18:24:32.096499 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:32.095632 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-kjc2z"] Sep 11 18:24:32.131132 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:32.131099 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-kjc2z"] Sep 11 18:24:32.131405 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:24:32.131382 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-cghx2], unattached volumes=[], failed to process volumes=[kube-api-access-cghx2]: context canceled" pod="kuadrant-system/authorino-557f876f69-kjc2z" podUID="a44ec470-1d3f-4883-860c-27046ba5edda" Sep 11 18:24:32.158805 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:32.158770 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-bkfhs"] Sep 11 18:24:32.161348 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:32.161325 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-bkfhs" Sep 11 18:24:32.163516 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:32.163490 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 11 18:24:32.170289 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:32.170251 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-bkfhs"] Sep 11 18:24:32.189965 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:32.189935 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cghx2\" (UniqueName: \"kubernetes.io/projected/a44ec470-1d3f-4883-860c-27046ba5edda-kube-api-access-cghx2\") pod \"authorino-557f876f69-kjc2z\" (UID: \"a44ec470-1d3f-4883-860c-27046ba5edda\") " pod="kuadrant-system/authorino-557f876f69-kjc2z" Sep 11 18:24:32.190105 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:32.189997 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kphd\" (UniqueName: \"kubernetes.io/projected/ac440c2a-2de5-41f6-ab55-f8c0713546c7-kube-api-access-2kphd\") pod \"authorino-6d4dfc649f-bkfhs\" (UID: \"ac440c2a-2de5-41f6-ab55-f8c0713546c7\") " pod="kuadrant-system/authorino-6d4dfc649f-bkfhs" Sep 11 18:24:32.190158 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:32.190129 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ac440c2a-2de5-41f6-ab55-f8c0713546c7-tls-cert\") pod \"authorino-6d4dfc649f-bkfhs\" (UID: \"ac440c2a-2de5-41f6-ab55-f8c0713546c7\") " pod="kuadrant-system/authorino-6d4dfc649f-bkfhs" Sep 11 18:24:32.290720 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:32.290681 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cghx2\" (UniqueName: \"kubernetes.io/projected/a44ec470-1d3f-4883-860c-27046ba5edda-kube-api-access-cghx2\") pod \"authorino-557f876f69-kjc2z\" (UID: \"a44ec470-1d3f-4883-860c-27046ba5edda\") " pod="kuadrant-system/authorino-557f876f69-kjc2z" Sep 11 18:24:32.290900 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:32.290747 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kphd\" (UniqueName: \"kubernetes.io/projected/ac440c2a-2de5-41f6-ab55-f8c0713546c7-kube-api-access-2kphd\") pod \"authorino-6d4dfc649f-bkfhs\" (UID: \"ac440c2a-2de5-41f6-ab55-f8c0713546c7\") " pod="kuadrant-system/authorino-6d4dfc649f-bkfhs" Sep 11 18:24:32.290900 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:32.290810 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ac440c2a-2de5-41f6-ab55-f8c0713546c7-tls-cert\") pod \"authorino-6d4dfc649f-bkfhs\" (UID: \"ac440c2a-2de5-41f6-ab55-f8c0713546c7\") " pod="kuadrant-system/authorino-6d4dfc649f-bkfhs" Sep 11 18:24:32.293436 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:32.293406 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ac440c2a-2de5-41f6-ab55-f8c0713546c7-tls-cert\") pod \"authorino-6d4dfc649f-bkfhs\" (UID: \"ac440c2a-2de5-41f6-ab55-f8c0713546c7\") " pod="kuadrant-system/authorino-6d4dfc649f-bkfhs" Sep 11 18:24:32.299550 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:32.299518 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cghx2\" (UniqueName: \"kubernetes.io/projected/a44ec470-1d3f-4883-860c-27046ba5edda-kube-api-access-cghx2\") pod \"authorino-557f876f69-kjc2z\" (UID: \"a44ec470-1d3f-4883-860c-27046ba5edda\") " pod="kuadrant-system/authorino-557f876f69-kjc2z" Sep 11 18:24:32.299634 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:32.299521 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kphd\" (UniqueName: \"kubernetes.io/projected/ac440c2a-2de5-41f6-ab55-f8c0713546c7-kube-api-access-2kphd\") pod \"authorino-6d4dfc649f-bkfhs\" (UID: \"ac440c2a-2de5-41f6-ab55-f8c0713546c7\") " pod="kuadrant-system/authorino-6d4dfc649f-bkfhs" Sep 11 18:24:32.308119 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:32.308094 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-bkfhs"] Sep 11 18:24:32.308436 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:32.308422 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-bkfhs" Sep 11 18:24:32.336208 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:32.336132 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-588f7bbdbc-xbgfd"] Sep 11 18:24:32.338945 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:32.338927 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-588f7bbdbc-xbgfd" Sep 11 18:24:32.350926 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:32.350893 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-588f7bbdbc-xbgfd"] Sep 11 18:24:32.392109 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:32.392024 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7fl9\" (UniqueName: \"kubernetes.io/projected/2c9447e1-49fa-4158-9867-a28502b5641d-kube-api-access-f7fl9\") pod \"authorino-588f7bbdbc-xbgfd\" (UID: \"2c9447e1-49fa-4158-9867-a28502b5641d\") " pod="kuadrant-system/authorino-588f7bbdbc-xbgfd" Sep 11 18:24:32.392274 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:32.392233 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2c9447e1-49fa-4158-9867-a28502b5641d-tls-cert\") pod \"authorino-588f7bbdbc-xbgfd\" (UID: \"2c9447e1-49fa-4158-9867-a28502b5641d\") " pod="kuadrant-system/authorino-588f7bbdbc-xbgfd" Sep 11 18:24:32.493518 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:32.493480 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7fl9\" (UniqueName: \"kubernetes.io/projected/2c9447e1-49fa-4158-9867-a28502b5641d-kube-api-access-f7fl9\") pod \"authorino-588f7bbdbc-xbgfd\" (UID: \"2c9447e1-49fa-4158-9867-a28502b5641d\") " pod="kuadrant-system/authorino-588f7bbdbc-xbgfd" Sep 11 18:24:32.493690 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:32.493556 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2c9447e1-49fa-4158-9867-a28502b5641d-tls-cert\") pod \"authorino-588f7bbdbc-xbgfd\" (UID: \"2c9447e1-49fa-4158-9867-a28502b5641d\") " pod="kuadrant-system/authorino-588f7bbdbc-xbgfd" Sep 11 18:24:32.495904 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:32.495872 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2c9447e1-49fa-4158-9867-a28502b5641d-tls-cert\") pod \"authorino-588f7bbdbc-xbgfd\" (UID: \"2c9447e1-49fa-4158-9867-a28502b5641d\") " pod="kuadrant-system/authorino-588f7bbdbc-xbgfd" Sep 11 18:24:32.502992 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:32.502968 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7fl9\" (UniqueName: \"kubernetes.io/projected/2c9447e1-49fa-4158-9867-a28502b5641d-kube-api-access-f7fl9\") pod \"authorino-588f7bbdbc-xbgfd\" (UID: \"2c9447e1-49fa-4158-9867-a28502b5641d\") " pod="kuadrant-system/authorino-588f7bbdbc-xbgfd" Sep 11 18:24:32.584413 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:32.584388 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-bkfhs"] Sep 11 18:24:32.586197 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:24:32.586171 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac440c2a_2de5_41f6_ab55_f8c0713546c7.slice/crio-1fc823aeb8b59b538532baf762f058cf9a52ee5cfb68b97013db5281cb42a3fd WatchSource:0}: Error finding container 1fc823aeb8b59b538532baf762f058cf9a52ee5cfb68b97013db5281cb42a3fd: Status 404 returned error can't find the container with id 1fc823aeb8b59b538532baf762f058cf9a52ee5cfb68b97013db5281cb42a3fd Sep 11 18:24:32.649047 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:32.649009 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-588f7bbdbc-xbgfd" Sep 11 18:24:32.812727 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:32.812694 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-kjc2z" Sep 11 18:24:32.812727 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:32.812707 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d4dfc649f-bkfhs" event={"ID":"ac440c2a-2de5-41f6-ab55-f8c0713546c7","Type":"ContainerStarted","Data":"1fc823aeb8b59b538532baf762f058cf9a52ee5cfb68b97013db5281cb42a3fd"} Sep 11 18:24:32.819870 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:32.819848 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-kjc2z" Sep 11 18:24:32.896366 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:32.896267 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cghx2\" (UniqueName: \"kubernetes.io/projected/a44ec470-1d3f-4883-860c-27046ba5edda-kube-api-access-cghx2\") pod \"a44ec470-1d3f-4883-860c-27046ba5edda\" (UID: \"a44ec470-1d3f-4883-860c-27046ba5edda\") " Sep 11 18:24:32.898453 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:32.898423 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a44ec470-1d3f-4883-860c-27046ba5edda-kube-api-access-cghx2" (OuterVolumeSpecName: "kube-api-access-cghx2") pod "a44ec470-1d3f-4883-860c-27046ba5edda" (UID: "a44ec470-1d3f-4883-860c-27046ba5edda"). InnerVolumeSpecName "kube-api-access-cghx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:24:32.918040 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:32.918015 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-588f7bbdbc-xbgfd"] Sep 11 18:24:32.919860 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:24:32.919835 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c9447e1_49fa_4158_9867_a28502b5641d.slice/crio-050bae03fec7f282290f63a5844de4ad24eb46ae189c349ac1106684417a915a WatchSource:0}: Error finding container 050bae03fec7f282290f63a5844de4ad24eb46ae189c349ac1106684417a915a: Status 404 returned error can't find the container with id 050bae03fec7f282290f63a5844de4ad24eb46ae189c349ac1106684417a915a Sep 11 18:24:32.997855 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:32.997828 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cghx2\" (UniqueName: \"kubernetes.io/projected/a44ec470-1d3f-4883-860c-27046ba5edda-kube-api-access-cghx2\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:24:33.820475 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:33.820431 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-588f7bbdbc-xbgfd" event={"ID":"2c9447e1-49fa-4158-9867-a28502b5641d","Type":"ContainerStarted","Data":"9555d8a728b8d1d296989ebb410a63fc0e68d45fe1489defa1a732520f3e878a"} Sep 11 18:24:33.820898 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:33.820485 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-588f7bbdbc-xbgfd" event={"ID":"2c9447e1-49fa-4158-9867-a28502b5641d","Type":"ContainerStarted","Data":"050bae03fec7f282290f63a5844de4ad24eb46ae189c349ac1106684417a915a"} Sep 11 18:24:33.821881 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:33.821854 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d4dfc649f-bkfhs" event={"ID":"ac440c2a-2de5-41f6-ab55-f8c0713546c7","Type":"ContainerStarted","Data":"71a772debb5a9007b6d3506455240378cb251c0f821a7e0073da21d71ce9c503"} Sep 11 18:24:33.822004 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:33.821884 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-kjc2z" Sep 11 18:24:33.822064 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:33.821884 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6d4dfc649f-bkfhs" podUID="ac440c2a-2de5-41f6-ab55-f8c0713546c7" containerName="authorino" containerID="cri-o://71a772debb5a9007b6d3506455240378cb251c0f821a7e0073da21d71ce9c503" gracePeriod=30 Sep 11 18:24:33.835952 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:33.835907 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-588f7bbdbc-xbgfd" podStartSLOduration=1.43700209 podStartE2EDuration="1.835890856s" totalImagesPullingTime="398.888766ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:24:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:24:33.83458044 +0000 UTC m=+555.331598194" watchObservedRunningTime="2026-09-11 18:24:33.835890856 +0000 UTC m=+555.332908610" Sep 11 18:24:33.852132 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:33.851796 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6d4dfc649f-bkfhs" podStartSLOduration=1.351141217 podStartE2EDuration="1.851774585s" totalImagesPullingTime="500.633368ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:24:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:24:33.85093896 +0000 UTC m=+555.347956705" watchObservedRunningTime="2026-09-11 18:24:33.851774585 +0000 UTC m=+555.348792340" Sep 11 18:24:33.872845 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:33.872810 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-qnmrd"] Sep 11 18:24:33.873126 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:33.873092 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-qnmrd" podUID="a7fb7861-dc94-40d9-ae45-047104357cfb" containerName="authorino" containerID="cri-o://97ac26558023f08c723bcdd50691bd18229b6e8b9d05d7559ae02ba506d73346" gracePeriod=30 Sep 11 18:24:33.876069 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:33.875999 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-kjc2z"] Sep 11 18:24:33.883380 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:33.883352 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-kjc2z"] Sep 11 18:24:34.131891 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:34.131864 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-bkfhs" Sep 11 18:24:34.136040 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:34.136021 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-qnmrd" Sep 11 18:24:34.209483 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:34.209452 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2kphd\" (UniqueName: \"kubernetes.io/projected/ac440c2a-2de5-41f6-ab55-f8c0713546c7-kube-api-access-2kphd\") pod \"ac440c2a-2de5-41f6-ab55-f8c0713546c7\" (UID: \"ac440c2a-2de5-41f6-ab55-f8c0713546c7\") " Sep 11 18:24:34.209627 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:34.209526 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s9dzk\" (UniqueName: \"kubernetes.io/projected/a7fb7861-dc94-40d9-ae45-047104357cfb-kube-api-access-s9dzk\") pod \"a7fb7861-dc94-40d9-ae45-047104357cfb\" (UID: \"a7fb7861-dc94-40d9-ae45-047104357cfb\") " Sep 11 18:24:34.209627 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:34.209573 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ac440c2a-2de5-41f6-ab55-f8c0713546c7-tls-cert\") pod \"ac440c2a-2de5-41f6-ab55-f8c0713546c7\" (UID: \"ac440c2a-2de5-41f6-ab55-f8c0713546c7\") " Sep 11 18:24:34.211642 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:34.211608 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7fb7861-dc94-40d9-ae45-047104357cfb-kube-api-access-s9dzk" (OuterVolumeSpecName: "kube-api-access-s9dzk") pod "a7fb7861-dc94-40d9-ae45-047104357cfb" (UID: "a7fb7861-dc94-40d9-ae45-047104357cfb"). InnerVolumeSpecName "kube-api-access-s9dzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:24:34.211768 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:34.211673 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac440c2a-2de5-41f6-ab55-f8c0713546c7-kube-api-access-2kphd" (OuterVolumeSpecName: "kube-api-access-2kphd") pod "ac440c2a-2de5-41f6-ab55-f8c0713546c7" (UID: "ac440c2a-2de5-41f6-ab55-f8c0713546c7"). InnerVolumeSpecName "kube-api-access-2kphd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:24:34.220114 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:34.220090 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac440c2a-2de5-41f6-ab55-f8c0713546c7-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "ac440c2a-2de5-41f6-ab55-f8c0713546c7" (UID: "ac440c2a-2de5-41f6-ab55-f8c0713546c7"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:24:34.310922 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:34.310888 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kphd\" (UniqueName: \"kubernetes.io/projected/ac440c2a-2de5-41f6-ab55-f8c0713546c7-kube-api-access-2kphd\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:24:34.310922 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:34.310920 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9dzk\" (UniqueName: \"kubernetes.io/projected/a7fb7861-dc94-40d9-ae45-047104357cfb-kube-api-access-s9dzk\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:24:34.311166 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:34.310929 2561 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ac440c2a-2de5-41f6-ab55-f8c0713546c7-tls-cert\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:24:34.828632 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:34.828591 2561 generic.go:356] "Generic (PLEG): container finished" podID="a7fb7861-dc94-40d9-ae45-047104357cfb" containerID="97ac26558023f08c723bcdd50691bd18229b6e8b9d05d7559ae02ba506d73346" exitCode=0 Sep 11 18:24:34.828632 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:34.828634 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-qnmrd" Sep 11 18:24:34.829183 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:34.828673 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-qnmrd" event={"ID":"a7fb7861-dc94-40d9-ae45-047104357cfb","Type":"ContainerDied","Data":"97ac26558023f08c723bcdd50691bd18229b6e8b9d05d7559ae02ba506d73346"} Sep 11 18:24:34.829183 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:34.828713 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-qnmrd" event={"ID":"a7fb7861-dc94-40d9-ae45-047104357cfb","Type":"ContainerDied","Data":"a23cbbe18013c276dd7f16aac35daa0fd27bed3d15fe7b276ea925c34f94a3a0"} Sep 11 18:24:34.829183 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:34.828730 2561 scope.go:117] "RemoveContainer" containerID="97ac26558023f08c723bcdd50691bd18229b6e8b9d05d7559ae02ba506d73346" Sep 11 18:24:34.830119 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:34.830044 2561 generic.go:356] "Generic (PLEG): container finished" podID="ac440c2a-2de5-41f6-ab55-f8c0713546c7" containerID="71a772debb5a9007b6d3506455240378cb251c0f821a7e0073da21d71ce9c503" exitCode=0 Sep 11 18:24:34.830119 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:34.830098 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-bkfhs" Sep 11 18:24:34.830257 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:34.830120 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d4dfc649f-bkfhs" event={"ID":"ac440c2a-2de5-41f6-ab55-f8c0713546c7","Type":"ContainerDied","Data":"71a772debb5a9007b6d3506455240378cb251c0f821a7e0073da21d71ce9c503"} Sep 11 18:24:34.830257 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:34.830156 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d4dfc649f-bkfhs" event={"ID":"ac440c2a-2de5-41f6-ab55-f8c0713546c7","Type":"ContainerDied","Data":"1fc823aeb8b59b538532baf762f058cf9a52ee5cfb68b97013db5281cb42a3fd"} Sep 11 18:24:34.837842 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:34.837824 2561 scope.go:117] "RemoveContainer" containerID="97ac26558023f08c723bcdd50691bd18229b6e8b9d05d7559ae02ba506d73346" Sep 11 18:24:34.838226 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:24:34.838186 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97ac26558023f08c723bcdd50691bd18229b6e8b9d05d7559ae02ba506d73346\": container with ID starting with 97ac26558023f08c723bcdd50691bd18229b6e8b9d05d7559ae02ba506d73346 not found: ID does not exist" containerID="97ac26558023f08c723bcdd50691bd18229b6e8b9d05d7559ae02ba506d73346" Sep 11 18:24:34.838336 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:34.838271 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97ac26558023f08c723bcdd50691bd18229b6e8b9d05d7559ae02ba506d73346"} err="failed to get container status \"97ac26558023f08c723bcdd50691bd18229b6e8b9d05d7559ae02ba506d73346\": rpc error: code = NotFound desc = could not find container \"97ac26558023f08c723bcdd50691bd18229b6e8b9d05d7559ae02ba506d73346\": container with ID starting with 97ac26558023f08c723bcdd50691bd18229b6e8b9d05d7559ae02ba506d73346 not found: ID does not exist" Sep 11 18:24:34.838336 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:34.838299 2561 scope.go:117] "RemoveContainer" containerID="71a772debb5a9007b6d3506455240378cb251c0f821a7e0073da21d71ce9c503" Sep 11 18:24:34.846968 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:34.846953 2561 scope.go:117] "RemoveContainer" containerID="71a772debb5a9007b6d3506455240378cb251c0f821a7e0073da21d71ce9c503" Sep 11 18:24:34.847237 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:24:34.847219 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71a772debb5a9007b6d3506455240378cb251c0f821a7e0073da21d71ce9c503\": container with ID starting with 71a772debb5a9007b6d3506455240378cb251c0f821a7e0073da21d71ce9c503 not found: ID does not exist" containerID="71a772debb5a9007b6d3506455240378cb251c0f821a7e0073da21d71ce9c503" Sep 11 18:24:34.847311 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:34.847247 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71a772debb5a9007b6d3506455240378cb251c0f821a7e0073da21d71ce9c503"} err="failed to get container status \"71a772debb5a9007b6d3506455240378cb251c0f821a7e0073da21d71ce9c503\": rpc error: code = NotFound desc = could not find container \"71a772debb5a9007b6d3506455240378cb251c0f821a7e0073da21d71ce9c503\": container with ID starting with 71a772debb5a9007b6d3506455240378cb251c0f821a7e0073da21d71ce9c503 not found: ID does not exist" Sep 11 18:24:34.854279 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:34.854256 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-qnmrd"] Sep 11 18:24:34.858683 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:34.858662 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-qnmrd"] Sep 11 18:24:34.869800 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:34.869777 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-bkfhs"] Sep 11 18:24:34.875038 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:34.875017 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-bkfhs"] Sep 11 18:24:35.139647 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:35.139568 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a44ec470-1d3f-4883-860c-27046ba5edda" path="/var/lib/kubelet/pods/a44ec470-1d3f-4883-860c-27046ba5edda/volumes" Sep 11 18:24:35.139805 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:35.139793 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7fb7861-dc94-40d9-ae45-047104357cfb" path="/var/lib/kubelet/pods/a7fb7861-dc94-40d9-ae45-047104357cfb/volumes" Sep 11 18:24:35.140143 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:24:35.140129 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac440c2a-2de5-41f6-ab55-f8c0713546c7" path="/var/lib/kubelet/pods/ac440c2a-2de5-41f6-ab55-f8c0713546c7/volumes" Sep 11 18:25:15.807470 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:15.807426 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-ldlb4"] Sep 11 18:25:15.807951 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:15.807845 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac440c2a-2de5-41f6-ab55-f8c0713546c7" containerName="authorino" Sep 11 18:25:15.807951 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:15.807858 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac440c2a-2de5-41f6-ab55-f8c0713546c7" containerName="authorino" Sep 11 18:25:15.807951 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:15.807876 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7fb7861-dc94-40d9-ae45-047104357cfb" containerName="authorino" Sep 11 18:25:15.807951 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:15.807882 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7fb7861-dc94-40d9-ae45-047104357cfb" containerName="authorino" Sep 11 18:25:15.807951 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:15.807941 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac440c2a-2de5-41f6-ab55-f8c0713546c7" containerName="authorino" Sep 11 18:25:15.807951 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:15.807954 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7fb7861-dc94-40d9-ae45-047104357cfb" containerName="authorino" Sep 11 18:25:15.810927 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:15.810910 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-ldlb4" Sep 11 18:25:15.813319 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:15.813295 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 11 18:25:15.821724 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:15.821698 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-ldlb4"] Sep 11 18:25:15.886588 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:15.886360 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf29bed6-97b3-4988-aa11-ffebe83a653a-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-ldlb4\" (UID: \"cf29bed6-97b3-4988-aa11-ffebe83a653a\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-ldlb4" Sep 11 18:25:15.886588 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:15.886420 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2kcg\" (UniqueName: \"kubernetes.io/projected/cf29bed6-97b3-4988-aa11-ffebe83a653a-kube-api-access-c2kcg\") pod \"payload-pre-processing-59fbc47bf8-ldlb4\" (UID: \"cf29bed6-97b3-4988-aa11-ffebe83a653a\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-ldlb4" Sep 11 18:25:15.987760 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:15.987620 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf29bed6-97b3-4988-aa11-ffebe83a653a-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-ldlb4\" (UID: \"cf29bed6-97b3-4988-aa11-ffebe83a653a\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-ldlb4" Sep 11 18:25:15.987962 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:15.987776 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2kcg\" (UniqueName: \"kubernetes.io/projected/cf29bed6-97b3-4988-aa11-ffebe83a653a-kube-api-access-c2kcg\") pod \"payload-pre-processing-59fbc47bf8-ldlb4\" (UID: \"cf29bed6-97b3-4988-aa11-ffebe83a653a\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-ldlb4" Sep 11 18:25:15.988598 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:15.988568 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf29bed6-97b3-4988-aa11-ffebe83a653a-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-ldlb4\" (UID: \"cf29bed6-97b3-4988-aa11-ffebe83a653a\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-ldlb4" Sep 11 18:25:16.000948 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:16.000912 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2kcg\" (UniqueName: \"kubernetes.io/projected/cf29bed6-97b3-4988-aa11-ffebe83a653a-kube-api-access-c2kcg\") pod \"payload-pre-processing-59fbc47bf8-ldlb4\" (UID: \"cf29bed6-97b3-4988-aa11-ffebe83a653a\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-ldlb4" Sep 11 18:25:16.123594 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:16.123499 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-ldlb4" Sep 11 18:25:16.399151 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:16.399119 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-ldlb4"] Sep 11 18:25:16.400452 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:25:16.400427 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf29bed6_97b3_4988_aa11_ffebe83a653a.slice/crio-15caf5ed94a517bd4c3b450cafad363f6fa1c4e78813b82fc384c78ad3e137fd WatchSource:0}: Error finding container 15caf5ed94a517bd4c3b450cafad363f6fa1c4e78813b82fc384c78ad3e137fd: Status 404 returned error can't find the container with id 15caf5ed94a517bd4c3b450cafad363f6fa1c4e78813b82fc384c78ad3e137fd Sep 11 18:25:17.070198 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:17.070157 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-ldlb4" event={"ID":"cf29bed6-97b3-4988-aa11-ffebe83a653a","Type":"ContainerStarted","Data":"15caf5ed94a517bd4c3b450cafad363f6fa1c4e78813b82fc384c78ad3e137fd"} Sep 11 18:25:18.078510 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:18.078402 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-ldlb4" event={"ID":"cf29bed6-97b3-4988-aa11-ffebe83a653a","Type":"ContainerStarted","Data":"cce07cebb797fc4eb0cb27b298f4785bcbe6e87010ff6357ffd6645bce6a604e"} Sep 11 18:25:18.078885 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:18.078575 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-ldlb4" Sep 11 18:25:18.099200 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:18.099114 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-ldlb4" podStartSLOduration=1.678566484 podStartE2EDuration="3.099090321s" totalImagesPullingTime="1.420523837s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:25:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:25:18.098243963 +0000 UTC m=+599.595261718" watchObservedRunningTime="2026-09-11 18:25:18.099090321 +0000 UTC m=+599.596108076" Sep 11 18:25:19.276807 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:19.276682 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dj524_60702ef3-98ef-463b-b877-3a00825353d8/console-operator/1.log" Sep 11 18:25:19.312791 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:19.276971 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dj524_60702ef3-98ef-463b-b877-3a00825353d8/console-operator/1.log" Sep 11 18:25:19.316074 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:19.316025 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4knd_42e5ddd8-a96c-45a1-adf9-caa47ee902db/ovn-acl-logging/0.log" Sep 11 18:25:19.316270 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:19.316246 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4knd_42e5ddd8-a96c-45a1-adf9-caa47ee902db/ovn-acl-logging/0.log" Sep 11 18:25:26.135379 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:26.135344 2561 generic.go:356] "Generic (PLEG): container finished" podID="09816140-ebbc-4f2d-9e1a-a11e86d773a4" containerID="7035135a4da19409fb74bb8b6e3c17a77f63c3e31a82d1e5df51fa919ac2e6f7" exitCode=0 Sep 11 18:25:26.135793 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:26.135419 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cbvsl" event={"ID":"09816140-ebbc-4f2d-9e1a-a11e86d773a4","Type":"ContainerDied","Data":"7035135a4da19409fb74bb8b6e3c17a77f63c3e31a82d1e5df51fa919ac2e6f7"} Sep 11 18:25:26.135836 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:26.135809 2561 scope.go:117] "RemoveContainer" containerID="7035135a4da19409fb74bb8b6e3c17a77f63c3e31a82d1e5df51fa919ac2e6f7" Sep 11 18:25:26.136747 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:26.136719 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:25:26.136803 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:26.136791 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:25:26.136838 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:26.136823 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:25:27.142842 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:27.142806 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cbvsl" event={"ID":"09816140-ebbc-4f2d-9e1a-a11e86d773a4","Type":"ContainerStarted","Data":"acb5f4dce3bbe25026a4c42b3468d4c5e933407454d6609c8af09ccdeb2fb09e"} Sep 11 18:25:28.141925 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:28.141880 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cbvsl" Sep 11 18:25:28.142777 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:28.142746 2561 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-cbvsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Sep 11 18:25:28.142910 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:28.142821 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cbvsl" podUID="09816140-ebbc-4f2d-9e1a-a11e86d773a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Sep 11 18:25:29.089324 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:29.089283 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-ldlb4" Sep 11 18:25:29.141998 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:29.141965 2561 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-cbvsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Sep 11 18:25:29.142183 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:29.142035 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cbvsl" podUID="09816140-ebbc-4f2d-9e1a-a11e86d773a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Sep 11 18:25:30.063410 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:30.063378 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mh9n8"] Sep 11 18:25:30.066732 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:30.066698 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mh9n8" Sep 11 18:25:30.069074 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:30.069032 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 11 18:25:30.069210 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:30.069091 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 11 18:25:30.069766 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:30.069742 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-zv74c\"" Sep 11 18:25:30.069876 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:30.069801 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Sep 11 18:25:30.078460 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:30.078430 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mh9n8"] Sep 11 18:25:30.132749 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:30.132711 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f4905e1-a9c3-42bd-9205-5c0d9c163e7c-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mh9n8\" (UID: \"6f4905e1-a9c3-42bd-9205-5c0d9c163e7c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mh9n8" Sep 11 18:25:30.132923 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:30.132752 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f4905e1-a9c3-42bd-9205-5c0d9c163e7c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mh9n8\" (UID: \"6f4905e1-a9c3-42bd-9205-5c0d9c163e7c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mh9n8" Sep 11 18:25:30.132923 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:30.132790 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f4905e1-a9c3-42bd-9205-5c0d9c163e7c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mh9n8\" (UID: \"6f4905e1-a9c3-42bd-9205-5c0d9c163e7c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mh9n8" Sep 11 18:25:30.132923 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:30.132835 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f4905e1-a9c3-42bd-9205-5c0d9c163e7c-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mh9n8\" (UID: \"6f4905e1-a9c3-42bd-9205-5c0d9c163e7c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mh9n8" Sep 11 18:25:30.132923 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:30.132882 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f4905e1-a9c3-42bd-9205-5c0d9c163e7c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mh9n8\" (UID: \"6f4905e1-a9c3-42bd-9205-5c0d9c163e7c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mh9n8" Sep 11 18:25:30.132923 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:30.132903 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ttrh\" (UniqueName: \"kubernetes.io/projected/6f4905e1-a9c3-42bd-9205-5c0d9c163e7c-kube-api-access-9ttrh\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mh9n8\" (UID: \"6f4905e1-a9c3-42bd-9205-5c0d9c163e7c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mh9n8" Sep 11 18:25:30.141884 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:30.141860 2561 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-cbvsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Sep 11 18:25:30.141948 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:30.141924 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cbvsl" podUID="09816140-ebbc-4f2d-9e1a-a11e86d773a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Sep 11 18:25:30.233891 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:30.233856 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f4905e1-a9c3-42bd-9205-5c0d9c163e7c-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mh9n8\" (UID: \"6f4905e1-a9c3-42bd-9205-5c0d9c163e7c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mh9n8" Sep 11 18:25:30.234099 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:30.233926 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f4905e1-a9c3-42bd-9205-5c0d9c163e7c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mh9n8\" (UID: \"6f4905e1-a9c3-42bd-9205-5c0d9c163e7c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mh9n8" Sep 11 18:25:30.234099 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:30.233946 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ttrh\" (UniqueName: \"kubernetes.io/projected/6f4905e1-a9c3-42bd-9205-5c0d9c163e7c-kube-api-access-9ttrh\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mh9n8\" (UID: \"6f4905e1-a9c3-42bd-9205-5c0d9c163e7c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mh9n8" Sep 11 18:25:30.234099 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:30.233974 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f4905e1-a9c3-42bd-9205-5c0d9c163e7c-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mh9n8\" (UID: \"6f4905e1-a9c3-42bd-9205-5c0d9c163e7c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mh9n8" Sep 11 18:25:30.234099 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:30.233993 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f4905e1-a9c3-42bd-9205-5c0d9c163e7c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mh9n8\" (UID: \"6f4905e1-a9c3-42bd-9205-5c0d9c163e7c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mh9n8" Sep 11 18:25:30.234099 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:30.234018 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f4905e1-a9c3-42bd-9205-5c0d9c163e7c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mh9n8\" (UID: \"6f4905e1-a9c3-42bd-9205-5c0d9c163e7c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mh9n8" Sep 11 18:25:30.234496 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:30.234461 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f4905e1-a9c3-42bd-9205-5c0d9c163e7c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mh9n8\" (UID: \"6f4905e1-a9c3-42bd-9205-5c0d9c163e7c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mh9n8" Sep 11 18:25:30.234602 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:30.234497 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f4905e1-a9c3-42bd-9205-5c0d9c163e7c-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mh9n8\" (UID: \"6f4905e1-a9c3-42bd-9205-5c0d9c163e7c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mh9n8" Sep 11 18:25:30.234602 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:30.234566 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f4905e1-a9c3-42bd-9205-5c0d9c163e7c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mh9n8\" (UID: \"6f4905e1-a9c3-42bd-9205-5c0d9c163e7c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mh9n8" Sep 11 18:25:30.236278 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:30.236253 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f4905e1-a9c3-42bd-9205-5c0d9c163e7c-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mh9n8\" (UID: \"6f4905e1-a9c3-42bd-9205-5c0d9c163e7c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mh9n8" Sep 11 18:25:30.236616 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:30.236594 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f4905e1-a9c3-42bd-9205-5c0d9c163e7c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mh9n8\" (UID: \"6f4905e1-a9c3-42bd-9205-5c0d9c163e7c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mh9n8" Sep 11 18:25:30.242516 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:30.242492 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ttrh\" (UniqueName: \"kubernetes.io/projected/6f4905e1-a9c3-42bd-9205-5c0d9c163e7c-kube-api-access-9ttrh\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mh9n8\" (UID: \"6f4905e1-a9c3-42bd-9205-5c0d9c163e7c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mh9n8" Sep 11 18:25:30.379910 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:30.379808 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mh9n8" Sep 11 18:25:30.673127 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:30.673088 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mh9n8"] Sep 11 18:25:30.674780 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:25:30.674749 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f4905e1_a9c3_42bd_9205_5c0d9c163e7c.slice/crio-2bc1f33b08d1b94933f542295d500bf33f984fb1b2f8709d8dc293046eca560f WatchSource:0}: Error finding container 2bc1f33b08d1b94933f542295d500bf33f984fb1b2f8709d8dc293046eca560f: Status 404 returned error can't find the container with id 2bc1f33b08d1b94933f542295d500bf33f984fb1b2f8709d8dc293046eca560f Sep 11 18:25:30.676633 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:30.676609 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 18:25:31.142200 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:31.142108 2561 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-cbvsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Sep 11 18:25:31.142200 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:31.142184 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cbvsl" podUID="09816140-ebbc-4f2d-9e1a-a11e86d773a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Sep 11 18:25:31.167409 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:31.167355 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mh9n8" event={"ID":"6f4905e1-a9c3-42bd-9205-5c0d9c163e7c","Type":"ContainerStarted","Data":"2bc1f33b08d1b94933f542295d500bf33f984fb1b2f8709d8dc293046eca560f"} Sep 11 18:25:32.141907 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:32.141875 2561 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-cbvsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Sep 11 18:25:32.142080 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:32.141940 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cbvsl" podUID="09816140-ebbc-4f2d-9e1a-a11e86d773a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Sep 11 18:25:32.632418 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:32.632367 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cbvsl" Sep 11 18:25:33.145126 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:33.145091 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cbvsl" Sep 11 18:25:33.181084 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:33.181028 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mh9n8" event={"ID":"6f4905e1-a9c3-42bd-9205-5c0d9c163e7c","Type":"ContainerStarted","Data":"bc49f70a2a4f834a03dedc67756d0aabe7ad010578cf3bb19da4d05353693849"} Sep 11 18:25:33.182102 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:33.182047 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cbvsl" Sep 11 18:25:33.198265 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:33.198215 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mh9n8" podStartSLOduration=1.734910756 podStartE2EDuration="3.198197326s" totalImagesPullingTime="1.46328657s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:25:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:25:33.19738873 +0000 UTC m=+614.694406484" watchObservedRunningTime="2026-09-11 18:25:33.198197326 +0000 UTC m=+614.695215109" Sep 11 18:25:40.380978 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:40.380937 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mh9n8" Sep 11 18:25:40.381468 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:40.381006 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mh9n8" Sep 11 18:25:40.393530 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:40.393499 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mh9n8" Sep 11 18:25:41.235541 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:41.235513 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mh9n8" Sep 11 18:25:46.140601 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:46.140552 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v9pzq"] Sep 11 18:25:46.143178 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:46.143153 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v9pzq" Sep 11 18:25:46.145505 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:46.145480 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Sep 11 18:25:46.154044 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:46.154013 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v9pzq"] Sep 11 18:25:46.289198 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:46.289166 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07e2555f-0967-4633-bc02-ad8262d1e749-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-v9pzq\" (UID: \"07e2555f-0967-4633-bc02-ad8262d1e749\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v9pzq" Sep 11 18:25:46.289198 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:46.289200 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07e2555f-0967-4633-bc02-ad8262d1e749-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-v9pzq\" (UID: \"07e2555f-0967-4633-bc02-ad8262d1e749\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v9pzq" Sep 11 18:25:46.289434 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:46.289226 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07e2555f-0967-4633-bc02-ad8262d1e749-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-v9pzq\" (UID: \"07e2555f-0967-4633-bc02-ad8262d1e749\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v9pzq" Sep 11 18:25:46.289434 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:46.289306 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7np4n\" (UniqueName: \"kubernetes.io/projected/07e2555f-0967-4633-bc02-ad8262d1e749-kube-api-access-7np4n\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-v9pzq\" (UID: \"07e2555f-0967-4633-bc02-ad8262d1e749\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v9pzq" Sep 11 18:25:46.289434 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:46.289343 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07e2555f-0967-4633-bc02-ad8262d1e749-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-v9pzq\" (UID: \"07e2555f-0967-4633-bc02-ad8262d1e749\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v9pzq" Sep 11 18:25:46.289546 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:46.289470 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07e2555f-0967-4633-bc02-ad8262d1e749-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-v9pzq\" (UID: \"07e2555f-0967-4633-bc02-ad8262d1e749\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v9pzq" Sep 11 18:25:46.391109 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:46.390997 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07e2555f-0967-4633-bc02-ad8262d1e749-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-v9pzq\" (UID: \"07e2555f-0967-4633-bc02-ad8262d1e749\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v9pzq" Sep 11 18:25:46.391109 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:46.391033 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07e2555f-0967-4633-bc02-ad8262d1e749-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-v9pzq\" (UID: \"07e2555f-0967-4633-bc02-ad8262d1e749\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v9pzq" Sep 11 18:25:46.391109 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:46.391079 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07e2555f-0967-4633-bc02-ad8262d1e749-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-v9pzq\" (UID: \"07e2555f-0967-4633-bc02-ad8262d1e749\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v9pzq" Sep 11 18:25:46.391393 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:46.391113 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7np4n\" (UniqueName: \"kubernetes.io/projected/07e2555f-0967-4633-bc02-ad8262d1e749-kube-api-access-7np4n\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-v9pzq\" (UID: \"07e2555f-0967-4633-bc02-ad8262d1e749\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v9pzq" Sep 11 18:25:46.391393 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:46.391137 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07e2555f-0967-4633-bc02-ad8262d1e749-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-v9pzq\" (UID: \"07e2555f-0967-4633-bc02-ad8262d1e749\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v9pzq" Sep 11 18:25:46.391393 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:46.391213 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07e2555f-0967-4633-bc02-ad8262d1e749-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-v9pzq\" (UID: \"07e2555f-0967-4633-bc02-ad8262d1e749\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v9pzq" Sep 11 18:25:46.391544 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:46.391510 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07e2555f-0967-4633-bc02-ad8262d1e749-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-v9pzq\" (UID: \"07e2555f-0967-4633-bc02-ad8262d1e749\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v9pzq" Sep 11 18:25:46.391681 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:46.391660 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07e2555f-0967-4633-bc02-ad8262d1e749-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-v9pzq\" (UID: \"07e2555f-0967-4633-bc02-ad8262d1e749\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v9pzq" Sep 11 18:25:46.391741 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:46.391660 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07e2555f-0967-4633-bc02-ad8262d1e749-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-v9pzq\" (UID: \"07e2555f-0967-4633-bc02-ad8262d1e749\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v9pzq" Sep 11 18:25:46.393547 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:46.393519 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07e2555f-0967-4633-bc02-ad8262d1e749-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-v9pzq\" (UID: \"07e2555f-0967-4633-bc02-ad8262d1e749\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v9pzq" Sep 11 18:25:46.393718 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:46.393700 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07e2555f-0967-4633-bc02-ad8262d1e749-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-v9pzq\" (UID: \"07e2555f-0967-4633-bc02-ad8262d1e749\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v9pzq" Sep 11 18:25:46.400237 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:46.400208 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7np4n\" (UniqueName: \"kubernetes.io/projected/07e2555f-0967-4633-bc02-ad8262d1e749-kube-api-access-7np4n\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-v9pzq\" (UID: \"07e2555f-0967-4633-bc02-ad8262d1e749\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v9pzq" Sep 11 18:25:46.458029 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:46.457984 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v9pzq" Sep 11 18:25:46.733047 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:46.733018 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v9pzq"] Sep 11 18:25:46.735642 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:25:46.735614 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07e2555f_0967_4633_bc02_ad8262d1e749.slice/crio-f6f117a9149f48145fc5841ff8acffa2a40b9489eb262c287c88bd27ff6769df WatchSource:0}: Error finding container f6f117a9149f48145fc5841ff8acffa2a40b9489eb262c287c88bd27ff6769df: Status 404 returned error can't find the container with id f6f117a9149f48145fc5841ff8acffa2a40b9489eb262c287c88bd27ff6769df Sep 11 18:25:47.263904 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:47.263863 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v9pzq" event={"ID":"07e2555f-0967-4633-bc02-ad8262d1e749","Type":"ContainerStarted","Data":"91f61d6d618eb5c9a8c66ed92ded27d51b337ed11cc634f8b49f7f95a23951b9"} Sep 11 18:25:47.264386 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:47.263923 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v9pzq" event={"ID":"07e2555f-0967-4633-bc02-ad8262d1e749","Type":"ContainerStarted","Data":"f6f117a9149f48145fc5841ff8acffa2a40b9489eb262c287c88bd27ff6769df"} Sep 11 18:25:47.283487 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:47.283420 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v9pzq" podStartSLOduration=1.10869371 podStartE2EDuration="1.283400092s" totalImagesPullingTime="174.706382ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:25:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:25:47.279950377 +0000 UTC m=+628.776968144" watchObservedRunningTime="2026-09-11 18:25:47.283400092 +0000 UTC m=+628.780417851" Sep 11 18:25:56.458951 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:56.458905 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v9pzq" Sep 11 18:25:56.459411 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:56.458966 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v9pzq" Sep 11 18:25:56.471612 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:56.471582 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v9pzq" Sep 11 18:25:57.348658 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:25:57.348625 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-v9pzq" Sep 11 18:26:12.026480 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:26:12.026434 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-4r97j"] Sep 11 18:26:12.032314 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:26:12.032290 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4r97j" Sep 11 18:26:12.034556 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:26:12.034531 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Sep 11 18:26:12.038899 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:26:12.038871 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-4r97j"] Sep 11 18:26:12.138872 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:26:12.138833 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cacdfa6-534e-4981-8189-f5fa8d0e2c97-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4r97j\" (UID: \"5cacdfa6-534e-4981-8189-f5fa8d0e2c97\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4r97j" Sep 11 18:26:12.139041 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:26:12.138899 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cacdfa6-534e-4981-8189-f5fa8d0e2c97-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4r97j\" (UID: \"5cacdfa6-534e-4981-8189-f5fa8d0e2c97\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4r97j" Sep 11 18:26:12.139041 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:26:12.138962 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cacdfa6-534e-4981-8189-f5fa8d0e2c97-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4r97j\" (UID: \"5cacdfa6-534e-4981-8189-f5fa8d0e2c97\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4r97j" Sep 11 18:26:12.139041 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:26:12.138990 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cacdfa6-534e-4981-8189-f5fa8d0e2c97-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4r97j\" (UID: \"5cacdfa6-534e-4981-8189-f5fa8d0e2c97\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4r97j" Sep 11 18:26:12.139041 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:26:12.139018 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cacdfa6-534e-4981-8189-f5fa8d0e2c97-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4r97j\" (UID: \"5cacdfa6-534e-4981-8189-f5fa8d0e2c97\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4r97j" Sep 11 18:26:12.139041 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:26:12.139039 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hngkc\" (UniqueName: \"kubernetes.io/projected/5cacdfa6-534e-4981-8189-f5fa8d0e2c97-kube-api-access-hngkc\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4r97j\" (UID: \"5cacdfa6-534e-4981-8189-f5fa8d0e2c97\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4r97j" Sep 11 18:26:12.240241 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:26:12.240192 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cacdfa6-534e-4981-8189-f5fa8d0e2c97-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4r97j\" (UID: \"5cacdfa6-534e-4981-8189-f5fa8d0e2c97\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4r97j" Sep 11 18:26:12.240241 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:26:12.240242 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cacdfa6-534e-4981-8189-f5fa8d0e2c97-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4r97j\" (UID: \"5cacdfa6-534e-4981-8189-f5fa8d0e2c97\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4r97j" Sep 11 18:26:12.240494 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:26:12.240363 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cacdfa6-534e-4981-8189-f5fa8d0e2c97-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4r97j\" (UID: \"5cacdfa6-534e-4981-8189-f5fa8d0e2c97\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4r97j" Sep 11 18:26:12.240494 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:26:12.240415 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hngkc\" (UniqueName: \"kubernetes.io/projected/5cacdfa6-534e-4981-8189-f5fa8d0e2c97-kube-api-access-hngkc\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4r97j\" (UID: \"5cacdfa6-534e-4981-8189-f5fa8d0e2c97\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4r97j" Sep 11 18:26:12.240639 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:26:12.240514 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cacdfa6-534e-4981-8189-f5fa8d0e2c97-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4r97j\" (UID: \"5cacdfa6-534e-4981-8189-f5fa8d0e2c97\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4r97j" Sep 11 18:26:12.240639 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:26:12.240612 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cacdfa6-534e-4981-8189-f5fa8d0e2c97-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4r97j\" (UID: \"5cacdfa6-534e-4981-8189-f5fa8d0e2c97\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4r97j" Sep 11 18:26:12.240735 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:26:12.240649 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cacdfa6-534e-4981-8189-f5fa8d0e2c97-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4r97j\" (UID: \"5cacdfa6-534e-4981-8189-f5fa8d0e2c97\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4r97j" Sep 11 18:26:12.240735 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:26:12.240686 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cacdfa6-534e-4981-8189-f5fa8d0e2c97-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4r97j\" (UID: \"5cacdfa6-534e-4981-8189-f5fa8d0e2c97\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4r97j" Sep 11 18:26:12.240962 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:26:12.240942 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cacdfa6-534e-4981-8189-f5fa8d0e2c97-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4r97j\" (UID: \"5cacdfa6-534e-4981-8189-f5fa8d0e2c97\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4r97j" Sep 11 18:26:12.242767 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:26:12.242736 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cacdfa6-534e-4981-8189-f5fa8d0e2c97-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4r97j\" (UID: \"5cacdfa6-534e-4981-8189-f5fa8d0e2c97\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4r97j" Sep 11 18:26:12.242936 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:26:12.242916 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cacdfa6-534e-4981-8189-f5fa8d0e2c97-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4r97j\" (UID: \"5cacdfa6-534e-4981-8189-f5fa8d0e2c97\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4r97j" Sep 11 18:26:12.249253 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:26:12.249219 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hngkc\" (UniqueName: \"kubernetes.io/projected/5cacdfa6-534e-4981-8189-f5fa8d0e2c97-kube-api-access-hngkc\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4r97j\" (UID: \"5cacdfa6-534e-4981-8189-f5fa8d0e2c97\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4r97j" Sep 11 18:26:12.346344 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:26:12.346252 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4r97j" Sep 11 18:26:12.628575 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:26:12.628544 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-4r97j"] Sep 11 18:26:12.629776 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:26:12.629749 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cacdfa6_534e_4981_8189_f5fa8d0e2c97.slice/crio-d8818e942dcd9dd3ab3ebb76d3ceeb869d9e487dfd376d984c395227784f3daf WatchSource:0}: Error finding container d8818e942dcd9dd3ab3ebb76d3ceeb869d9e487dfd376d984c395227784f3daf: Status 404 returned error can't find the container with id d8818e942dcd9dd3ab3ebb76d3ceeb869d9e487dfd376d984c395227784f3daf Sep 11 18:26:13.428555 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:26:13.428516 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4r97j" event={"ID":"5cacdfa6-534e-4981-8189-f5fa8d0e2c97","Type":"ContainerStarted","Data":"ec9f27660f9ddc5ad4522276c7023ccdf0ceefaf74465281ba0c768c29b79974"} Sep 11 18:26:13.428961 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:26:13.428584 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4r97j" event={"ID":"5cacdfa6-534e-4981-8189-f5fa8d0e2c97","Type":"ContainerStarted","Data":"d8818e942dcd9dd3ab3ebb76d3ceeb869d9e487dfd376d984c395227784f3daf"} Sep 11 18:26:13.444980 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:26:13.444926 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4r97j" podStartSLOduration=1.244235178 podStartE2EDuration="1.444905038s" totalImagesPullingTime="200.66986ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:26:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:26:13.444440691 +0000 UTC m=+654.941458446" watchObservedRunningTime="2026-09-11 18:26:13.444905038 +0000 UTC m=+654.941922793" Sep 11 18:26:22.346894 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:26:22.346849 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4r97j" Sep 11 18:26:22.347415 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:26:22.346925 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4r97j" Sep 11 18:26:22.360041 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:26:22.360015 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4r97j" Sep 11 18:26:22.493230 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:26:22.493191 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4r97j" Sep 11 18:27:12.383088 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:12.382844 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5cc6c58786-qn445"] Sep 11 18:27:12.386832 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:12.386803 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5cc6c58786-qn445" Sep 11 18:27:12.396238 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:12.396207 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5cc6c58786-qn445"] Sep 11 18:27:12.503335 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:12.503298 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6923f940-4731-40d1-be20-530994fb7394-tls-cert\") pod \"authorino-5cc6c58786-qn445\" (UID: \"6923f940-4731-40d1-be20-530994fb7394\") " pod="kuadrant-system/authorino-5cc6c58786-qn445" Sep 11 18:27:12.503512 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:12.503358 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d7rn\" (UniqueName: \"kubernetes.io/projected/6923f940-4731-40d1-be20-530994fb7394-kube-api-access-7d7rn\") pod \"authorino-5cc6c58786-qn445\" (UID: \"6923f940-4731-40d1-be20-530994fb7394\") " pod="kuadrant-system/authorino-5cc6c58786-qn445" Sep 11 18:27:12.604888 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:12.604849 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d7rn\" (UniqueName: \"kubernetes.io/projected/6923f940-4731-40d1-be20-530994fb7394-kube-api-access-7d7rn\") pod \"authorino-5cc6c58786-qn445\" (UID: \"6923f940-4731-40d1-be20-530994fb7394\") " pod="kuadrant-system/authorino-5cc6c58786-qn445" Sep 11 18:27:12.605123 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:12.605003 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6923f940-4731-40d1-be20-530994fb7394-tls-cert\") pod \"authorino-5cc6c58786-qn445\" (UID: \"6923f940-4731-40d1-be20-530994fb7394\") " pod="kuadrant-system/authorino-5cc6c58786-qn445" Sep 11 18:27:12.607701 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:12.607671 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6923f940-4731-40d1-be20-530994fb7394-tls-cert\") pod \"authorino-5cc6c58786-qn445\" (UID: \"6923f940-4731-40d1-be20-530994fb7394\") " pod="kuadrant-system/authorino-5cc6c58786-qn445" Sep 11 18:27:12.613320 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:12.613296 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d7rn\" (UniqueName: \"kubernetes.io/projected/6923f940-4731-40d1-be20-530994fb7394-kube-api-access-7d7rn\") pod \"authorino-5cc6c58786-qn445\" (UID: \"6923f940-4731-40d1-be20-530994fb7394\") " pod="kuadrant-system/authorino-5cc6c58786-qn445" Sep 11 18:27:12.697923 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:12.697882 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5cc6c58786-qn445" Sep 11 18:27:12.972040 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:12.972003 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5cc6c58786-qn445"] Sep 11 18:27:12.973386 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:27:12.973357 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6923f940_4731_40d1_be20_530994fb7394.slice/crio-b03dce6fb3ca7ece87d9d14639b36b4129f094c60e366a05e32d9d3015e78a8b WatchSource:0}: Error finding container b03dce6fb3ca7ece87d9d14639b36b4129f094c60e366a05e32d9d3015e78a8b: Status 404 returned error can't find the container with id b03dce6fb3ca7ece87d9d14639b36b4129f094c60e366a05e32d9d3015e78a8b Sep 11 18:27:13.780422 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:13.780389 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5cc6c58786-qn445" event={"ID":"6923f940-4731-40d1-be20-530994fb7394","Type":"ContainerStarted","Data":"b03dce6fb3ca7ece87d9d14639b36b4129f094c60e366a05e32d9d3015e78a8b"} Sep 11 18:27:14.792328 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:14.792281 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5cc6c58786-qn445" event={"ID":"6923f940-4731-40d1-be20-530994fb7394","Type":"ContainerStarted","Data":"dd9e1d233442c9bc1dbb96e584c3ebad87559d935022a842c9a43d6954cefa1e"} Sep 11 18:27:14.809923 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:14.809847 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5cc6c58786-qn445" podStartSLOduration=1.963885285 podStartE2EDuration="2.80981204s" totalImagesPullingTime="845.926755ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:27:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:27:14.806826492 +0000 UTC m=+716.303844238" watchObservedRunningTime="2026-09-11 18:27:14.80981204 +0000 UTC m=+716.306829795" Sep 11 18:27:14.845580 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:14.845527 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-588f7bbdbc-xbgfd"] Sep 11 18:27:14.846354 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:14.846048 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-588f7bbdbc-xbgfd" podUID="2c9447e1-49fa-4158-9867-a28502b5641d" containerName="authorino" containerID="cri-o://9555d8a728b8d1d296989ebb410a63fc0e68d45fe1489defa1a732520f3e878a" gracePeriod=30 Sep 11 18:27:15.106460 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:15.106429 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-588f7bbdbc-xbgfd" Sep 11 18:27:15.232247 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:15.232211 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2c9447e1-49fa-4158-9867-a28502b5641d-tls-cert\") pod \"2c9447e1-49fa-4158-9867-a28502b5641d\" (UID: \"2c9447e1-49fa-4158-9867-a28502b5641d\") " Sep 11 18:27:15.232412 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:15.232295 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f7fl9\" (UniqueName: \"kubernetes.io/projected/2c9447e1-49fa-4158-9867-a28502b5641d-kube-api-access-f7fl9\") pod \"2c9447e1-49fa-4158-9867-a28502b5641d\" (UID: \"2c9447e1-49fa-4158-9867-a28502b5641d\") " Sep 11 18:27:15.234490 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:15.234449 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c9447e1-49fa-4158-9867-a28502b5641d-kube-api-access-f7fl9" (OuterVolumeSpecName: "kube-api-access-f7fl9") pod "2c9447e1-49fa-4158-9867-a28502b5641d" (UID: "2c9447e1-49fa-4158-9867-a28502b5641d"). InnerVolumeSpecName "kube-api-access-f7fl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:27:15.244822 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:15.244781 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c9447e1-49fa-4158-9867-a28502b5641d-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "2c9447e1-49fa-4158-9867-a28502b5641d" (UID: "2c9447e1-49fa-4158-9867-a28502b5641d"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:27:15.333615 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:15.333571 2561 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2c9447e1-49fa-4158-9867-a28502b5641d-tls-cert\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:27:15.333615 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:15.333609 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7fl9\" (UniqueName: \"kubernetes.io/projected/2c9447e1-49fa-4158-9867-a28502b5641d-kube-api-access-f7fl9\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:27:15.799309 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:15.799271 2561 generic.go:356] "Generic (PLEG): container finished" podID="2c9447e1-49fa-4158-9867-a28502b5641d" containerID="9555d8a728b8d1d296989ebb410a63fc0e68d45fe1489defa1a732520f3e878a" exitCode=0 Sep 11 18:27:15.799720 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:15.799331 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-588f7bbdbc-xbgfd" Sep 11 18:27:15.799720 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:15.799353 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-588f7bbdbc-xbgfd" event={"ID":"2c9447e1-49fa-4158-9867-a28502b5641d","Type":"ContainerDied","Data":"9555d8a728b8d1d296989ebb410a63fc0e68d45fe1489defa1a732520f3e878a"} Sep 11 18:27:15.799720 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:15.799400 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-588f7bbdbc-xbgfd" event={"ID":"2c9447e1-49fa-4158-9867-a28502b5641d","Type":"ContainerDied","Data":"050bae03fec7f282290f63a5844de4ad24eb46ae189c349ac1106684417a915a"} Sep 11 18:27:15.799720 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:15.799420 2561 scope.go:117] "RemoveContainer" containerID="9555d8a728b8d1d296989ebb410a63fc0e68d45fe1489defa1a732520f3e878a" Sep 11 18:27:15.809549 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:15.809521 2561 scope.go:117] "RemoveContainer" containerID="9555d8a728b8d1d296989ebb410a63fc0e68d45fe1489defa1a732520f3e878a" Sep 11 18:27:15.809888 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:27:15.809861 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9555d8a728b8d1d296989ebb410a63fc0e68d45fe1489defa1a732520f3e878a\": container with ID starting with 9555d8a728b8d1d296989ebb410a63fc0e68d45fe1489defa1a732520f3e878a not found: ID does not exist" containerID="9555d8a728b8d1d296989ebb410a63fc0e68d45fe1489defa1a732520f3e878a" Sep 11 18:27:15.809959 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:15.809895 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9555d8a728b8d1d296989ebb410a63fc0e68d45fe1489defa1a732520f3e878a"} err="failed to get container status \"9555d8a728b8d1d296989ebb410a63fc0e68d45fe1489defa1a732520f3e878a\": rpc error: code = NotFound desc = could not find container \"9555d8a728b8d1d296989ebb410a63fc0e68d45fe1489defa1a732520f3e878a\": container with ID starting with 9555d8a728b8d1d296989ebb410a63fc0e68d45fe1489defa1a732520f3e878a not found: ID does not exist" Sep 11 18:27:15.824998 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:15.824954 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-588f7bbdbc-xbgfd"] Sep 11 18:27:15.827456 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:15.827427 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-588f7bbdbc-xbgfd"] Sep 11 18:27:17.139544 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:17.139504 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c9447e1-49fa-4158-9867-a28502b5641d" path="/var/lib/kubelet/pods/2c9447e1-49fa-4158-9867-a28502b5641d/volumes" Sep 11 18:27:32.095533 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.095498 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s"] Sep 11 18:27:32.096040 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.095896 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c9447e1-49fa-4158-9867-a28502b5641d" containerName="authorino" Sep 11 18:27:32.096040 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.095908 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c9447e1-49fa-4158-9867-a28502b5641d" containerName="authorino" Sep 11 18:27:32.096040 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.095986 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c9447e1-49fa-4158-9867-a28502b5641d" containerName="authorino" Sep 11 18:27:32.100573 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.100549 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s" Sep 11 18:27:32.103690 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.103645 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-7d7509-openshift-default-dockercfg-8z789\"" Sep 11 18:27:32.118650 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.118616 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s"] Sep 11 18:27:32.159519 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.159198 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm"] Sep 11 18:27:32.163545 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.163517 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm" Sep 11 18:27:32.168189 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.168158 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-efa8b9-openshift-default-dockercfg-pvw2t\"" Sep 11 18:27:32.178336 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.178201 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm"] Sep 11 18:27:32.202833 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.202795 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/891f8576-360b-4004-b612-2169d1323649-credential-socket\") pod \"e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm\" (UID: \"891f8576-360b-4004-b612-2169d1323649\") " pod="openshift-ingress/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm" Sep 11 18:27:32.203150 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.203125 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11283c37-ec93-4ff1-a82b-ecc812c41b3d-workload-certs\") pod \"e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s\" (UID: \"11283c37-ec93-4ff1-a82b-ecc812c41b3d\") " pod="openshift-ingress/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s" Sep 11 18:27:32.203340 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.203294 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11283c37-ec93-4ff1-a82b-ecc812c41b3d-istio-token\") pod \"e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s\" (UID: \"11283c37-ec93-4ff1-a82b-ecc812c41b3d\") " pod="openshift-ingress/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s" Sep 11 18:27:32.203431 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.203411 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11283c37-ec93-4ff1-a82b-ecc812c41b3d-istiod-ca-cert\") pod \"e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s\" (UID: \"11283c37-ec93-4ff1-a82b-ecc812c41b3d\") " pod="openshift-ingress/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s" Sep 11 18:27:32.203494 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.203456 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11283c37-ec93-4ff1-a82b-ecc812c41b3d-istio-data\") pod \"e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s\" (UID: \"11283c37-ec93-4ff1-a82b-ecc812c41b3d\") " pod="openshift-ingress/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s" Sep 11 18:27:32.203544 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.203505 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/891f8576-360b-4004-b612-2169d1323649-workload-socket\") pod \"e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm\" (UID: \"891f8576-360b-4004-b612-2169d1323649\") " pod="openshift-ingress/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm" Sep 11 18:27:32.203834 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.203812 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsdth\" (UniqueName: \"kubernetes.io/projected/11283c37-ec93-4ff1-a82b-ecc812c41b3d-kube-api-access-tsdth\") pod \"e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s\" (UID: \"11283c37-ec93-4ff1-a82b-ecc812c41b3d\") " pod="openshift-ingress/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s" Sep 11 18:27:32.204017 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.203972 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11283c37-ec93-4ff1-a82b-ecc812c41b3d-workload-socket\") pod \"e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s\" (UID: \"11283c37-ec93-4ff1-a82b-ecc812c41b3d\") " pod="openshift-ingress/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s" Sep 11 18:27:32.204176 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.204021 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdvjd\" (UniqueName: \"kubernetes.io/projected/891f8576-360b-4004-b612-2169d1323649-kube-api-access-vdvjd\") pod \"e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm\" (UID: \"891f8576-360b-4004-b612-2169d1323649\") " pod="openshift-ingress/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm" Sep 11 18:27:32.204332 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.204306 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11283c37-ec93-4ff1-a82b-ecc812c41b3d-istio-podinfo\") pod \"e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s\" (UID: \"11283c37-ec93-4ff1-a82b-ecc812c41b3d\") " pod="openshift-ingress/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s" Sep 11 18:27:32.204399 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.204345 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/891f8576-360b-4004-b612-2169d1323649-workload-certs\") pod \"e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm\" (UID: \"891f8576-360b-4004-b612-2169d1323649\") " pod="openshift-ingress/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm" Sep 11 18:27:32.204457 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.204404 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/891f8576-360b-4004-b612-2169d1323649-istiod-ca-cert\") pod \"e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm\" (UID: \"891f8576-360b-4004-b612-2169d1323649\") " pod="openshift-ingress/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm" Sep 11 18:27:32.204457 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.204439 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/891f8576-360b-4004-b612-2169d1323649-istio-envoy\") pod \"e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm\" (UID: \"891f8576-360b-4004-b612-2169d1323649\") " pod="openshift-ingress/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm" Sep 11 18:27:32.204582 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.204465 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11283c37-ec93-4ff1-a82b-ecc812c41b3d-istio-envoy\") pod \"e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s\" (UID: \"11283c37-ec93-4ff1-a82b-ecc812c41b3d\") " pod="openshift-ingress/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s" Sep 11 18:27:32.204743 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.204721 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/891f8576-360b-4004-b612-2169d1323649-istio-podinfo\") pod \"e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm\" (UID: \"891f8576-360b-4004-b612-2169d1323649\") " pod="openshift-ingress/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm" Sep 11 18:27:32.204808 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.204774 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/891f8576-360b-4004-b612-2169d1323649-istio-token\") pod \"e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm\" (UID: \"891f8576-360b-4004-b612-2169d1323649\") " pod="openshift-ingress/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm" Sep 11 18:27:32.204808 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.204800 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/891f8576-360b-4004-b612-2169d1323649-istio-data\") pod \"e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm\" (UID: \"891f8576-360b-4004-b612-2169d1323649\") " pod="openshift-ingress/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm" Sep 11 18:27:32.204963 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.204948 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11283c37-ec93-4ff1-a82b-ecc812c41b3d-credential-socket\") pod \"e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s\" (UID: \"11283c37-ec93-4ff1-a82b-ecc812c41b3d\") " pod="openshift-ingress/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s" Sep 11 18:27:32.306294 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.306239 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11283c37-ec93-4ff1-a82b-ecc812c41b3d-istio-data\") pod \"e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s\" (UID: \"11283c37-ec93-4ff1-a82b-ecc812c41b3d\") " pod="openshift-ingress/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s" Sep 11 18:27:32.306476 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.306299 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/891f8576-360b-4004-b612-2169d1323649-workload-socket\") pod \"e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm\" (UID: \"891f8576-360b-4004-b612-2169d1323649\") " pod="openshift-ingress/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm" Sep 11 18:27:32.306476 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.306333 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsdth\" (UniqueName: \"kubernetes.io/projected/11283c37-ec93-4ff1-a82b-ecc812c41b3d-kube-api-access-tsdth\") pod \"e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s\" (UID: \"11283c37-ec93-4ff1-a82b-ecc812c41b3d\") " pod="openshift-ingress/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s" Sep 11 18:27:32.306476 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.306385 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11283c37-ec93-4ff1-a82b-ecc812c41b3d-workload-socket\") pod \"e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s\" (UID: \"11283c37-ec93-4ff1-a82b-ecc812c41b3d\") " pod="openshift-ingress/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s" Sep 11 18:27:32.306476 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.306411 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdvjd\" (UniqueName: \"kubernetes.io/projected/891f8576-360b-4004-b612-2169d1323649-kube-api-access-vdvjd\") pod \"e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm\" (UID: \"891f8576-360b-4004-b612-2169d1323649\") " pod="openshift-ingress/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm" Sep 11 18:27:32.306476 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.306447 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11283c37-ec93-4ff1-a82b-ecc812c41b3d-istio-podinfo\") pod \"e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s\" (UID: \"11283c37-ec93-4ff1-a82b-ecc812c41b3d\") " pod="openshift-ingress/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s" Sep 11 18:27:32.306782 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.306476 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/891f8576-360b-4004-b612-2169d1323649-workload-certs\") pod \"e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm\" (UID: \"891f8576-360b-4004-b612-2169d1323649\") " pod="openshift-ingress/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm" Sep 11 18:27:32.306782 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.306503 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/891f8576-360b-4004-b612-2169d1323649-istiod-ca-cert\") pod \"e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm\" (UID: \"891f8576-360b-4004-b612-2169d1323649\") " pod="openshift-ingress/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm" Sep 11 18:27:32.306782 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.306528 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/891f8576-360b-4004-b612-2169d1323649-istio-envoy\") pod \"e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm\" (UID: \"891f8576-360b-4004-b612-2169d1323649\") " pod="openshift-ingress/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm" Sep 11 18:27:32.306782 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.306556 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11283c37-ec93-4ff1-a82b-ecc812c41b3d-istio-envoy\") pod \"e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s\" (UID: \"11283c37-ec93-4ff1-a82b-ecc812c41b3d\") " pod="openshift-ingress/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s" Sep 11 18:27:32.306782 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.306593 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/891f8576-360b-4004-b612-2169d1323649-istio-podinfo\") pod \"e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm\" (UID: \"891f8576-360b-4004-b612-2169d1323649\") " pod="openshift-ingress/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm" Sep 11 18:27:32.306782 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.306628 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/891f8576-360b-4004-b612-2169d1323649-istio-token\") pod \"e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm\" (UID: \"891f8576-360b-4004-b612-2169d1323649\") " pod="openshift-ingress/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm" Sep 11 18:27:32.306782 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.306658 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/891f8576-360b-4004-b612-2169d1323649-istio-data\") pod \"e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm\" (UID: \"891f8576-360b-4004-b612-2169d1323649\") " pod="openshift-ingress/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm" Sep 11 18:27:32.306782 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.306702 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11283c37-ec93-4ff1-a82b-ecc812c41b3d-credential-socket\") pod \"e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s\" (UID: \"11283c37-ec93-4ff1-a82b-ecc812c41b3d\") " pod="openshift-ingress/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s" Sep 11 18:27:32.306782 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.306708 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11283c37-ec93-4ff1-a82b-ecc812c41b3d-istio-data\") pod \"e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s\" (UID: \"11283c37-ec93-4ff1-a82b-ecc812c41b3d\") " pod="openshift-ingress/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s" Sep 11 18:27:32.306782 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.306737 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/891f8576-360b-4004-b612-2169d1323649-credential-socket\") pod \"e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm\" (UID: \"891f8576-360b-4004-b612-2169d1323649\") " pod="openshift-ingress/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm" Sep 11 18:27:32.307332 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.306787 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11283c37-ec93-4ff1-a82b-ecc812c41b3d-workload-certs\") pod \"e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s\" (UID: \"11283c37-ec93-4ff1-a82b-ecc812c41b3d\") " pod="openshift-ingress/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s" Sep 11 18:27:32.307332 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.306818 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11283c37-ec93-4ff1-a82b-ecc812c41b3d-istio-token\") pod \"e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s\" (UID: \"11283c37-ec93-4ff1-a82b-ecc812c41b3d\") " pod="openshift-ingress/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s" Sep 11 18:27:32.307332 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.306845 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11283c37-ec93-4ff1-a82b-ecc812c41b3d-istiod-ca-cert\") pod \"e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s\" (UID: \"11283c37-ec93-4ff1-a82b-ecc812c41b3d\") " pod="openshift-ingress/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s" Sep 11 18:27:32.307492 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.307380 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/891f8576-360b-4004-b612-2169d1323649-istiod-ca-cert\") pod \"e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm\" (UID: \"891f8576-360b-4004-b612-2169d1323649\") " pod="openshift-ingress/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm" Sep 11 18:27:32.307585 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.307564 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11283c37-ec93-4ff1-a82b-ecc812c41b3d-istiod-ca-cert\") pod \"e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s\" (UID: \"11283c37-ec93-4ff1-a82b-ecc812c41b3d\") " pod="openshift-ingress/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s" Sep 11 18:27:32.307664 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.307646 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/891f8576-360b-4004-b612-2169d1323649-workload-socket\") pod \"e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm\" (UID: \"891f8576-360b-4004-b612-2169d1323649\") " pod="openshift-ingress/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm" Sep 11 18:27:32.308192 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.308164 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11283c37-ec93-4ff1-a82b-ecc812c41b3d-workload-socket\") pod \"e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s\" (UID: \"11283c37-ec93-4ff1-a82b-ecc812c41b3d\") " pod="openshift-ingress/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s" Sep 11 18:27:32.308889 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.308399 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/891f8576-360b-4004-b612-2169d1323649-istio-data\") pod \"e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm\" (UID: \"891f8576-360b-4004-b612-2169d1323649\") " pod="openshift-ingress/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm" Sep 11 18:27:32.308889 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.308655 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/891f8576-360b-4004-b612-2169d1323649-credential-socket\") pod \"e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm\" (UID: \"891f8576-360b-4004-b612-2169d1323649\") " pod="openshift-ingress/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm" Sep 11 18:27:32.308889 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.308767 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11283c37-ec93-4ff1-a82b-ecc812c41b3d-workload-certs\") pod \"e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s\" (UID: \"11283c37-ec93-4ff1-a82b-ecc812c41b3d\") " pod="openshift-ingress/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s" Sep 11 18:27:32.308889 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.308860 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11283c37-ec93-4ff1-a82b-ecc812c41b3d-credential-socket\") pod \"e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s\" (UID: \"11283c37-ec93-4ff1-a82b-ecc812c41b3d\") " pod="openshift-ingress/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s" Sep 11 18:27:32.309496 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.309475 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/891f8576-360b-4004-b612-2169d1323649-workload-certs\") pod \"e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm\" (UID: \"891f8576-360b-4004-b612-2169d1323649\") " pod="openshift-ingress/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm" Sep 11 18:27:32.310487 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.310465 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/891f8576-360b-4004-b612-2169d1323649-istio-envoy\") pod \"e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm\" (UID: \"891f8576-360b-4004-b612-2169d1323649\") " pod="openshift-ingress/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm" Sep 11 18:27:32.310817 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.310768 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11283c37-ec93-4ff1-a82b-ecc812c41b3d-istio-envoy\") pod \"e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s\" (UID: \"11283c37-ec93-4ff1-a82b-ecc812c41b3d\") " pod="openshift-ingress/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s" Sep 11 18:27:32.311282 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.311261 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/891f8576-360b-4004-b612-2169d1323649-istio-podinfo\") pod \"e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm\" (UID: \"891f8576-360b-4004-b612-2169d1323649\") " pod="openshift-ingress/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm" Sep 11 18:27:32.311939 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.311918 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11283c37-ec93-4ff1-a82b-ecc812c41b3d-istio-podinfo\") pod \"e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s\" (UID: \"11283c37-ec93-4ff1-a82b-ecc812c41b3d\") " pod="openshift-ingress/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s" Sep 11 18:27:32.317798 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.317767 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11283c37-ec93-4ff1-a82b-ecc812c41b3d-istio-token\") pod \"e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s\" (UID: \"11283c37-ec93-4ff1-a82b-ecc812c41b3d\") " pod="openshift-ingress/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s" Sep 11 18:27:32.318741 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.318681 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsdth\" (UniqueName: \"kubernetes.io/projected/11283c37-ec93-4ff1-a82b-ecc812c41b3d-kube-api-access-tsdth\") pod \"e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s\" (UID: \"11283c37-ec93-4ff1-a82b-ecc812c41b3d\") " pod="openshift-ingress/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s" Sep 11 18:27:32.323886 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.323495 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/891f8576-360b-4004-b612-2169d1323649-istio-token\") pod \"e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm\" (UID: \"891f8576-360b-4004-b612-2169d1323649\") " pod="openshift-ingress/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm" Sep 11 18:27:32.325691 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.325337 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdvjd\" (UniqueName: \"kubernetes.io/projected/891f8576-360b-4004-b612-2169d1323649-kube-api-access-vdvjd\") pod \"e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm\" (UID: \"891f8576-360b-4004-b612-2169d1323649\") " pod="openshift-ingress/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm" Sep 11 18:27:32.412302 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.412260 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s" Sep 11 18:27:32.478872 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.478627 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm" Sep 11 18:27:32.695984 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.695948 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s"] Sep 11 18:27:32.697571 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:27:32.697533 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11283c37_ec93_4ff1_a82b_ecc812c41b3d.slice/crio-6f63f3e97743ec53fab855d0965b87c649982cee4e9ba8ea008b13bc13107ecc WatchSource:0}: Error finding container 6f63f3e97743ec53fab855d0965b87c649982cee4e9ba8ea008b13bc13107ecc: Status 404 returned error can't find the container with id 6f63f3e97743ec53fab855d0965b87c649982cee4e9ba8ea008b13bc13107ecc Sep 11 18:27:32.700424 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.700378 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:27:32.700541 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.700478 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:27:32.700541 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.700527 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:27:32.795959 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.795930 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm"] Sep 11 18:27:32.797790 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:27:32.797759 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod891f8576_360b_4004_b612_2169d1323649.slice/crio-a2e3a32b355e18cb481e56708a64c8050cfc8da199c6ff0b4919d3ccbd0033b6 WatchSource:0}: Error finding container a2e3a32b355e18cb481e56708a64c8050cfc8da199c6ff0b4919d3ccbd0033b6: Status 404 returned error can't find the container with id a2e3a32b355e18cb481e56708a64c8050cfc8da199c6ff0b4919d3ccbd0033b6 Sep 11 18:27:32.800747 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.800706 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:27:32.800834 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.800794 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:27:32.800875 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.800834 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:27:32.910259 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.910218 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s" event={"ID":"11283c37-ec93-4ff1-a82b-ecc812c41b3d","Type":"ContainerStarted","Data":"4d7d1845627fb9b2afcf35ce69e88bae290d6c42957bb9c7184ae3f53c2bf247"} Sep 11 18:27:32.910424 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.910286 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s" event={"ID":"11283c37-ec93-4ff1-a82b-ecc812c41b3d","Type":"ContainerStarted","Data":"6f63f3e97743ec53fab855d0965b87c649982cee4e9ba8ea008b13bc13107ecc"} Sep 11 18:27:32.913262 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.912847 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm" event={"ID":"891f8576-360b-4004-b612-2169d1323649","Type":"ContainerStarted","Data":"7f53dc95d5b8ed078e3362f1f885c8b4f0ba38761ee23614e8b82934138015f7"} Sep 11 18:27:32.913262 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.912906 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm" event={"ID":"891f8576-360b-4004-b612-2169d1323649","Type":"ContainerStarted","Data":"a2e3a32b355e18cb481e56708a64c8050cfc8da199c6ff0b4919d3ccbd0033b6"} Sep 11 18:27:32.929985 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:32.929929 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s" podStartSLOduration=0.9299141 podStartE2EDuration="929.9141ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:27:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:27:32.928727509 +0000 UTC m=+734.425745265" watchObservedRunningTime="2026-09-11 18:27:32.9299141 +0000 UTC m=+734.426931854" Sep 11 18:27:33.412937 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:33.412892 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s" Sep 11 18:27:33.418077 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:33.418037 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s" Sep 11 18:27:33.444047 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:33.443990 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm" podStartSLOduration=1.443973127 podStartE2EDuration="1.443973127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:27:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:27:32.949910082 +0000 UTC m=+734.446927838" watchObservedRunningTime="2026-09-11 18:27:33.443973127 +0000 UTC m=+734.940990907" Sep 11 18:27:33.479473 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:33.479441 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm" Sep 11 18:27:33.485196 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:33.485171 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm" Sep 11 18:27:33.919139 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:33.919103 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm" Sep 11 18:27:33.919325 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:33.919163 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s" Sep 11 18:27:33.919914 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:33.919890 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm" Sep 11 18:27:33.920016 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:33.919947 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s" Sep 11 18:27:40.448321 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:40.448286 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn"] Sep 11 18:27:40.453347 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:40.453323 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn" Sep 11 18:27:40.456374 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:40.456347 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-7e8ab9-openshift-default-dockercfg-hfxb4\"" Sep 11 18:27:40.465284 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:40.465251 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn"] Sep 11 18:27:40.588309 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:40.588271 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3730018e-ef6b-4210-9007-0861b85202df-workload-socket\") pod \"e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn\" (UID: \"3730018e-ef6b-4210-9007-0861b85202df\") " pod="openshift-ingress/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn" Sep 11 18:27:40.588309 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:40.588310 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3730018e-ef6b-4210-9007-0861b85202df-istiod-ca-cert\") pod \"e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn\" (UID: \"3730018e-ef6b-4210-9007-0861b85202df\") " pod="openshift-ingress/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn" Sep 11 18:27:40.588585 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:40.588404 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3730018e-ef6b-4210-9007-0861b85202df-istio-data\") pod \"e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn\" (UID: \"3730018e-ef6b-4210-9007-0861b85202df\") " pod="openshift-ingress/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn" Sep 11 18:27:40.588585 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:40.588443 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3730018e-ef6b-4210-9007-0861b85202df-istio-token\") pod \"e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn\" (UID: \"3730018e-ef6b-4210-9007-0861b85202df\") " pod="openshift-ingress/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn" Sep 11 18:27:40.588585 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:40.588469 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3730018e-ef6b-4210-9007-0861b85202df-istio-envoy\") pod \"e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn\" (UID: \"3730018e-ef6b-4210-9007-0861b85202df\") " pod="openshift-ingress/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn" Sep 11 18:27:40.588585 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:40.588505 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78f9q\" (UniqueName: \"kubernetes.io/projected/3730018e-ef6b-4210-9007-0861b85202df-kube-api-access-78f9q\") pod \"e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn\" (UID: \"3730018e-ef6b-4210-9007-0861b85202df\") " pod="openshift-ingress/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn" Sep 11 18:27:40.588805 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:40.588629 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3730018e-ef6b-4210-9007-0861b85202df-workload-certs\") pod \"e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn\" (UID: \"3730018e-ef6b-4210-9007-0861b85202df\") " pod="openshift-ingress/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn" Sep 11 18:27:40.588805 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:40.588672 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3730018e-ef6b-4210-9007-0861b85202df-istio-podinfo\") pod \"e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn\" (UID: \"3730018e-ef6b-4210-9007-0861b85202df\") " pod="openshift-ingress/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn" Sep 11 18:27:40.588805 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:40.588712 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3730018e-ef6b-4210-9007-0861b85202df-credential-socket\") pod \"e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn\" (UID: \"3730018e-ef6b-4210-9007-0861b85202df\") " pod="openshift-ingress/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn" Sep 11 18:27:40.690070 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:40.690019 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3730018e-ef6b-4210-9007-0861b85202df-workload-socket\") pod \"e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn\" (UID: \"3730018e-ef6b-4210-9007-0861b85202df\") " pod="openshift-ingress/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn" Sep 11 18:27:40.690070 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:40.690073 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3730018e-ef6b-4210-9007-0861b85202df-istiod-ca-cert\") pod \"e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn\" (UID: \"3730018e-ef6b-4210-9007-0861b85202df\") " pod="openshift-ingress/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn" Sep 11 18:27:40.690305 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:40.690119 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3730018e-ef6b-4210-9007-0861b85202df-istio-data\") pod \"e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn\" (UID: \"3730018e-ef6b-4210-9007-0861b85202df\") " pod="openshift-ingress/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn" Sep 11 18:27:40.690305 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:40.690146 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3730018e-ef6b-4210-9007-0861b85202df-istio-token\") pod \"e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn\" (UID: \"3730018e-ef6b-4210-9007-0861b85202df\") " pod="openshift-ingress/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn" Sep 11 18:27:40.690305 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:40.690173 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3730018e-ef6b-4210-9007-0861b85202df-istio-envoy\") pod \"e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn\" (UID: \"3730018e-ef6b-4210-9007-0861b85202df\") " pod="openshift-ingress/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn" Sep 11 18:27:40.690305 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:40.690208 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78f9q\" (UniqueName: \"kubernetes.io/projected/3730018e-ef6b-4210-9007-0861b85202df-kube-api-access-78f9q\") pod \"e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn\" (UID: \"3730018e-ef6b-4210-9007-0861b85202df\") " pod="openshift-ingress/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn" Sep 11 18:27:40.690305 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:40.690254 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3730018e-ef6b-4210-9007-0861b85202df-workload-certs\") pod \"e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn\" (UID: \"3730018e-ef6b-4210-9007-0861b85202df\") " pod="openshift-ingress/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn" Sep 11 18:27:40.690305 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:40.690287 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3730018e-ef6b-4210-9007-0861b85202df-istio-podinfo\") pod \"e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn\" (UID: \"3730018e-ef6b-4210-9007-0861b85202df\") " pod="openshift-ingress/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn" Sep 11 18:27:40.690619 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:40.690323 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3730018e-ef6b-4210-9007-0861b85202df-credential-socket\") pod \"e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn\" (UID: \"3730018e-ef6b-4210-9007-0861b85202df\") " pod="openshift-ingress/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn" Sep 11 18:27:40.690619 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:40.690517 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3730018e-ef6b-4210-9007-0861b85202df-workload-socket\") pod \"e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn\" (UID: \"3730018e-ef6b-4210-9007-0861b85202df\") " pod="openshift-ingress/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn" Sep 11 18:27:40.690619 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:40.690564 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3730018e-ef6b-4210-9007-0861b85202df-istio-data\") pod \"e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn\" (UID: \"3730018e-ef6b-4210-9007-0861b85202df\") " pod="openshift-ingress/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn" Sep 11 18:27:40.691041 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:40.691012 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3730018e-ef6b-4210-9007-0861b85202df-workload-certs\") pod \"e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn\" (UID: \"3730018e-ef6b-4210-9007-0861b85202df\") " pod="openshift-ingress/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn" Sep 11 18:27:40.691198 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:40.691088 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3730018e-ef6b-4210-9007-0861b85202df-credential-socket\") pod \"e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn\" (UID: \"3730018e-ef6b-4210-9007-0861b85202df\") " pod="openshift-ingress/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn" Sep 11 18:27:40.691406 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:40.691384 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3730018e-ef6b-4210-9007-0861b85202df-istiod-ca-cert\") pod \"e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn\" (UID: \"3730018e-ef6b-4210-9007-0861b85202df\") " pod="openshift-ingress/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn" Sep 11 18:27:40.692725 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:40.692702 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3730018e-ef6b-4210-9007-0861b85202df-istio-envoy\") pod \"e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn\" (UID: \"3730018e-ef6b-4210-9007-0861b85202df\") " pod="openshift-ingress/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn" Sep 11 18:27:40.693121 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:40.693099 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3730018e-ef6b-4210-9007-0861b85202df-istio-podinfo\") pod \"e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn\" (UID: \"3730018e-ef6b-4210-9007-0861b85202df\") " pod="openshift-ingress/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn" Sep 11 18:27:40.699705 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:40.699642 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78f9q\" (UniqueName: \"kubernetes.io/projected/3730018e-ef6b-4210-9007-0861b85202df-kube-api-access-78f9q\") pod \"e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn\" (UID: \"3730018e-ef6b-4210-9007-0861b85202df\") " pod="openshift-ingress/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn" Sep 11 18:27:40.699705 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:40.699654 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3730018e-ef6b-4210-9007-0861b85202df-istio-token\") pod \"e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn\" (UID: \"3730018e-ef6b-4210-9007-0861b85202df\") " pod="openshift-ingress/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn" Sep 11 18:27:40.766474 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:40.766428 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn" Sep 11 18:27:40.791900 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:40.791852 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c"] Sep 11 18:27:40.797616 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:40.797588 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c" Sep 11 18:27:40.803426 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:40.800825 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-8370956b-openshift-default-dockercfg-mcbsz\"" Sep 11 18:27:40.812703 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:40.812285 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c"] Sep 11 18:27:40.892986 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:40.892943 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90875473-2bb0-471b-97a4-467f6b10a468-workload-certs\") pod \"e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c\" (UID: \"90875473-2bb0-471b-97a4-467f6b10a468\") " pod="openshift-ingress/e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c" Sep 11 18:27:40.893285 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:40.893011 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90875473-2bb0-471b-97a4-467f6b10a468-istiod-ca-cert\") pod \"e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c\" (UID: \"90875473-2bb0-471b-97a4-467f6b10a468\") " pod="openshift-ingress/e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c" Sep 11 18:27:40.893285 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:40.893065 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90875473-2bb0-471b-97a4-467f6b10a468-credential-socket\") pod \"e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c\" (UID: \"90875473-2bb0-471b-97a4-467f6b10a468\") " pod="openshift-ingress/e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c" Sep 11 18:27:40.893285 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:40.893133 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90875473-2bb0-471b-97a4-467f6b10a468-istio-token\") pod \"e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c\" (UID: \"90875473-2bb0-471b-97a4-467f6b10a468\") " pod="openshift-ingress/e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c" Sep 11 18:27:40.893285 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:40.893162 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkhns\" (UniqueName: \"kubernetes.io/projected/90875473-2bb0-471b-97a4-467f6b10a468-kube-api-access-nkhns\") pod \"e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c\" (UID: \"90875473-2bb0-471b-97a4-467f6b10a468\") " pod="openshift-ingress/e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c" Sep 11 18:27:40.893285 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:40.893199 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90875473-2bb0-471b-97a4-467f6b10a468-istio-podinfo\") pod \"e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c\" (UID: \"90875473-2bb0-471b-97a4-467f6b10a468\") " pod="openshift-ingress/e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c" Sep 11 18:27:40.893285 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:40.893283 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90875473-2bb0-471b-97a4-467f6b10a468-istio-data\") pod \"e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c\" (UID: \"90875473-2bb0-471b-97a4-467f6b10a468\") " pod="openshift-ingress/e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c" Sep 11 18:27:40.893741 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:40.893336 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90875473-2bb0-471b-97a4-467f6b10a468-workload-socket\") pod \"e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c\" (UID: \"90875473-2bb0-471b-97a4-467f6b10a468\") " pod="openshift-ingress/e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c" Sep 11 18:27:40.893741 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:40.893377 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90875473-2bb0-471b-97a4-467f6b10a468-istio-envoy\") pod \"e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c\" (UID: \"90875473-2bb0-471b-97a4-467f6b10a468\") " pod="openshift-ingress/e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c" Sep 11 18:27:40.994718 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:40.994617 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90875473-2bb0-471b-97a4-467f6b10a468-istio-data\") pod \"e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c\" (UID: \"90875473-2bb0-471b-97a4-467f6b10a468\") " pod="openshift-ingress/e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c" Sep 11 18:27:40.994718 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:40.994687 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90875473-2bb0-471b-97a4-467f6b10a468-workload-socket\") pod \"e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c\" (UID: \"90875473-2bb0-471b-97a4-467f6b10a468\") " pod="openshift-ingress/e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c" Sep 11 18:27:40.994718 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:40.994711 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90875473-2bb0-471b-97a4-467f6b10a468-istio-envoy\") pod \"e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c\" (UID: \"90875473-2bb0-471b-97a4-467f6b10a468\") " pod="openshift-ingress/e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c" Sep 11 18:27:40.994999 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:40.994752 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90875473-2bb0-471b-97a4-467f6b10a468-workload-certs\") pod \"e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c\" (UID: \"90875473-2bb0-471b-97a4-467f6b10a468\") " pod="openshift-ingress/e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c" Sep 11 18:27:40.994999 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:40.994793 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90875473-2bb0-471b-97a4-467f6b10a468-istiod-ca-cert\") pod \"e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c\" (UID: \"90875473-2bb0-471b-97a4-467f6b10a468\") " pod="openshift-ingress/e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c" Sep 11 18:27:40.994999 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:40.994829 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90875473-2bb0-471b-97a4-467f6b10a468-credential-socket\") pod \"e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c\" (UID: \"90875473-2bb0-471b-97a4-467f6b10a468\") " pod="openshift-ingress/e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c" Sep 11 18:27:40.994999 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:40.994896 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90875473-2bb0-471b-97a4-467f6b10a468-istio-token\") pod \"e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c\" (UID: \"90875473-2bb0-471b-97a4-467f6b10a468\") " pod="openshift-ingress/e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c" Sep 11 18:27:40.994999 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:40.994927 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkhns\" (UniqueName: \"kubernetes.io/projected/90875473-2bb0-471b-97a4-467f6b10a468-kube-api-access-nkhns\") pod \"e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c\" (UID: \"90875473-2bb0-471b-97a4-467f6b10a468\") " pod="openshift-ingress/e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c" Sep 11 18:27:40.994999 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:40.994970 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90875473-2bb0-471b-97a4-467f6b10a468-istio-podinfo\") pod \"e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c\" (UID: \"90875473-2bb0-471b-97a4-467f6b10a468\") " pod="openshift-ingress/e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c" Sep 11 18:27:40.995350 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:40.995185 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90875473-2bb0-471b-97a4-467f6b10a468-workload-socket\") pod \"e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c\" (UID: \"90875473-2bb0-471b-97a4-467f6b10a468\") " pod="openshift-ingress/e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c" Sep 11 18:27:40.995350 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:40.995196 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90875473-2bb0-471b-97a4-467f6b10a468-workload-certs\") pod \"e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c\" (UID: \"90875473-2bb0-471b-97a4-467f6b10a468\") " pod="openshift-ingress/e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c" Sep 11 18:27:40.995350 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:40.995253 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90875473-2bb0-471b-97a4-467f6b10a468-istio-data\") pod \"e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c\" (UID: \"90875473-2bb0-471b-97a4-467f6b10a468\") " pod="openshift-ingress/e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c" Sep 11 18:27:40.995502 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:40.995467 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90875473-2bb0-471b-97a4-467f6b10a468-credential-socket\") pod \"e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c\" (UID: \"90875473-2bb0-471b-97a4-467f6b10a468\") " pod="openshift-ingress/e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c" Sep 11 18:27:40.995762 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:40.995737 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90875473-2bb0-471b-97a4-467f6b10a468-istiod-ca-cert\") pod \"e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c\" (UID: \"90875473-2bb0-471b-97a4-467f6b10a468\") " pod="openshift-ingress/e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c" Sep 11 18:27:40.997594 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:40.997570 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90875473-2bb0-471b-97a4-467f6b10a468-istio-podinfo\") pod \"e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c\" (UID: \"90875473-2bb0-471b-97a4-467f6b10a468\") " pod="openshift-ingress/e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c" Sep 11 18:27:40.997594 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:40.997587 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90875473-2bb0-471b-97a4-467f6b10a468-istio-envoy\") pod \"e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c\" (UID: \"90875473-2bb0-471b-97a4-467f6b10a468\") " pod="openshift-ingress/e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c" Sep 11 18:27:41.004033 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:41.004011 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkhns\" (UniqueName: \"kubernetes.io/projected/90875473-2bb0-471b-97a4-467f6b10a468-kube-api-access-nkhns\") pod \"e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c\" (UID: \"90875473-2bb0-471b-97a4-467f6b10a468\") " pod="openshift-ingress/e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c" Sep 11 18:27:41.004183 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:41.004153 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90875473-2bb0-471b-97a4-467f6b10a468-istio-token\") pod \"e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c\" (UID: \"90875473-2bb0-471b-97a4-467f6b10a468\") " pod="openshift-ingress/e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c" Sep 11 18:27:41.068243 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:41.068211 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn"] Sep 11 18:27:41.069117 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:27:41.069088 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3730018e_ef6b_4210_9007_0861b85202df.slice/crio-d2084a2870191eaf91fa9c528b9b3ae2a4d345c23f0141ccc38d3f34e3a7c2dd WatchSource:0}: Error finding container d2084a2870191eaf91fa9c528b9b3ae2a4d345c23f0141ccc38d3f34e3a7c2dd: Status 404 returned error can't find the container with id d2084a2870191eaf91fa9c528b9b3ae2a4d345c23f0141ccc38d3f34e3a7c2dd Sep 11 18:27:41.071425 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:41.071387 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:27:41.071548 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:41.071485 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:27:41.071548 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:41.071535 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:27:41.118801 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:41.118770 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c" Sep 11 18:27:41.408475 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:41.408437 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c"] Sep 11 18:27:41.409628 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:27:41.409588 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90875473_2bb0_471b_97a4_467f6b10a468.slice/crio-01a7dfd0ace3af1703b013bc17f28f68a7ae11cbd2c0a474c23f6f3069d05011 WatchSource:0}: Error finding container 01a7dfd0ace3af1703b013bc17f28f68a7ae11cbd2c0a474c23f6f3069d05011: Status 404 returned error can't find the container with id 01a7dfd0ace3af1703b013bc17f28f68a7ae11cbd2c0a474c23f6f3069d05011 Sep 11 18:27:41.412048 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:41.412011 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:27:41.412180 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:41.412096 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:27:41.412180 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:41.412127 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:27:41.969761 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:41.969711 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c" event={"ID":"90875473-2bb0-471b-97a4-467f6b10a468","Type":"ContainerStarted","Data":"466b8eae0deb060375948d4b461fb26118e3a59cd89efb40875656195860f1c2"} Sep 11 18:27:41.970246 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:41.969783 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c" event={"ID":"90875473-2bb0-471b-97a4-467f6b10a468","Type":"ContainerStarted","Data":"01a7dfd0ace3af1703b013bc17f28f68a7ae11cbd2c0a474c23f6f3069d05011"} Sep 11 18:27:41.971447 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:41.971418 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn" event={"ID":"3730018e-ef6b-4210-9007-0861b85202df","Type":"ContainerStarted","Data":"966aca739243da692077f59bc34833a01cdffbc4013aa25842441a919628b33d"} Sep 11 18:27:41.971570 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:41.971468 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn" event={"ID":"3730018e-ef6b-4210-9007-0861b85202df","Type":"ContainerStarted","Data":"d2084a2870191eaf91fa9c528b9b3ae2a4d345c23f0141ccc38d3f34e3a7c2dd"} Sep 11 18:27:41.989602 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:41.989543 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c" podStartSLOduration=1.9895211910000001 podStartE2EDuration="1.989521191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:27:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:27:41.988384345 +0000 UTC m=+743.485402125" watchObservedRunningTime="2026-09-11 18:27:41.989521191 +0000 UTC m=+743.486538945" Sep 11 18:27:42.009962 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:42.009906 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn" podStartSLOduration=2.009890688 podStartE2EDuration="2.009890688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:27:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:27:42.007568517 +0000 UTC m=+743.504586271" watchObservedRunningTime="2026-09-11 18:27:42.009890688 +0000 UTC m=+743.506908442" Sep 11 18:27:42.119564 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:42.119523 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c" Sep 11 18:27:42.138902 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:42.138865 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c" Sep 11 18:27:42.766781 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:42.766741 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn" Sep 11 18:27:42.772225 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:42.772188 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn" Sep 11 18:27:42.988075 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:42.987537 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn" Sep 11 18:27:42.988075 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:42.987595 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c" Sep 11 18:27:42.993943 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:42.993224 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c" Sep 11 18:27:42.993943 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:42.993623 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn" Sep 11 18:27:50.280939 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:50.280897 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd"] Sep 11 18:27:50.307032 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:50.306997 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd"] Sep 11 18:27:50.307439 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:50.307367 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd" Sep 11 18:27:50.314355 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:50.314326 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-1de19d-openshift-default-dockercfg-cpgn2\"" Sep 11 18:27:50.413079 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:50.399768 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e9b9a66-30ad-4d94-95aa-d09ee8eea072-istio-podinfo\") pod \"e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd\" (UID: \"7e9b9a66-30ad-4d94-95aa-d09ee8eea072\") " pod="openshift-ingress/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd" Sep 11 18:27:50.413079 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:50.399836 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e9b9a66-30ad-4d94-95aa-d09ee8eea072-istio-envoy\") pod \"e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd\" (UID: \"7e9b9a66-30ad-4d94-95aa-d09ee8eea072\") " pod="openshift-ingress/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd" Sep 11 18:27:50.413079 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:50.399916 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e9b9a66-30ad-4d94-95aa-d09ee8eea072-credential-socket\") pod \"e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd\" (UID: \"7e9b9a66-30ad-4d94-95aa-d09ee8eea072\") " pod="openshift-ingress/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd" Sep 11 18:27:50.413079 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:50.399957 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5kg5\" (UniqueName: \"kubernetes.io/projected/7e9b9a66-30ad-4d94-95aa-d09ee8eea072-kube-api-access-k5kg5\") pod \"e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd\" (UID: \"7e9b9a66-30ad-4d94-95aa-d09ee8eea072\") " pod="openshift-ingress/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd" Sep 11 18:27:50.413079 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:50.399987 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e9b9a66-30ad-4d94-95aa-d09ee8eea072-workload-socket\") pod \"e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd\" (UID: \"7e9b9a66-30ad-4d94-95aa-d09ee8eea072\") " pod="openshift-ingress/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd" Sep 11 18:27:50.413079 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:50.400011 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e9b9a66-30ad-4d94-95aa-d09ee8eea072-istio-token\") pod \"e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd\" (UID: \"7e9b9a66-30ad-4d94-95aa-d09ee8eea072\") " pod="openshift-ingress/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd" Sep 11 18:27:50.413079 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:50.400039 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e9b9a66-30ad-4d94-95aa-d09ee8eea072-istio-data\") pod \"e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd\" (UID: \"7e9b9a66-30ad-4d94-95aa-d09ee8eea072\") " pod="openshift-ingress/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd" Sep 11 18:27:50.413079 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:50.400104 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e9b9a66-30ad-4d94-95aa-d09ee8eea072-istiod-ca-cert\") pod \"e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd\" (UID: \"7e9b9a66-30ad-4d94-95aa-d09ee8eea072\") " pod="openshift-ingress/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd" Sep 11 18:27:50.413079 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:50.400148 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e9b9a66-30ad-4d94-95aa-d09ee8eea072-workload-certs\") pod \"e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd\" (UID: \"7e9b9a66-30ad-4d94-95aa-d09ee8eea072\") " pod="openshift-ingress/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd" Sep 11 18:27:50.501822 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:50.501771 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e9b9a66-30ad-4d94-95aa-d09ee8eea072-credential-socket\") pod \"e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd\" (UID: \"7e9b9a66-30ad-4d94-95aa-d09ee8eea072\") " pod="openshift-ingress/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd" Sep 11 18:27:50.502027 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:50.501843 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5kg5\" (UniqueName: \"kubernetes.io/projected/7e9b9a66-30ad-4d94-95aa-d09ee8eea072-kube-api-access-k5kg5\") pod \"e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd\" (UID: \"7e9b9a66-30ad-4d94-95aa-d09ee8eea072\") " pod="openshift-ingress/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd" Sep 11 18:27:50.502027 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:50.501874 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e9b9a66-30ad-4d94-95aa-d09ee8eea072-workload-socket\") pod \"e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd\" (UID: \"7e9b9a66-30ad-4d94-95aa-d09ee8eea072\") " pod="openshift-ingress/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd" Sep 11 18:27:50.502027 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:50.501901 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e9b9a66-30ad-4d94-95aa-d09ee8eea072-istio-token\") pod \"e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd\" (UID: \"7e9b9a66-30ad-4d94-95aa-d09ee8eea072\") " pod="openshift-ingress/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd" Sep 11 18:27:50.502027 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:50.501926 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e9b9a66-30ad-4d94-95aa-d09ee8eea072-istio-data\") pod \"e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd\" (UID: \"7e9b9a66-30ad-4d94-95aa-d09ee8eea072\") " pod="openshift-ingress/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd" Sep 11 18:27:50.502027 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:50.501972 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e9b9a66-30ad-4d94-95aa-d09ee8eea072-istiod-ca-cert\") pod \"e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd\" (UID: \"7e9b9a66-30ad-4d94-95aa-d09ee8eea072\") " pod="openshift-ingress/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd" Sep 11 18:27:50.502027 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:50.502017 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e9b9a66-30ad-4d94-95aa-d09ee8eea072-workload-certs\") pod \"e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd\" (UID: \"7e9b9a66-30ad-4d94-95aa-d09ee8eea072\") " pod="openshift-ingress/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd" Sep 11 18:27:50.502404 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:50.502121 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e9b9a66-30ad-4d94-95aa-d09ee8eea072-istio-podinfo\") pod \"e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd\" (UID: \"7e9b9a66-30ad-4d94-95aa-d09ee8eea072\") " pod="openshift-ingress/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd" Sep 11 18:27:50.502404 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:50.502173 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e9b9a66-30ad-4d94-95aa-d09ee8eea072-istio-envoy\") pod \"e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd\" (UID: \"7e9b9a66-30ad-4d94-95aa-d09ee8eea072\") " pod="openshift-ingress/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd" Sep 11 18:27:50.503932 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:50.503296 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e9b9a66-30ad-4d94-95aa-d09ee8eea072-credential-socket\") pod \"e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd\" (UID: \"7e9b9a66-30ad-4d94-95aa-d09ee8eea072\") " pod="openshift-ingress/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd" Sep 11 18:27:50.503932 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:50.503400 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e9b9a66-30ad-4d94-95aa-d09ee8eea072-workload-socket\") pod \"e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd\" (UID: \"7e9b9a66-30ad-4d94-95aa-d09ee8eea072\") " pod="openshift-ingress/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd" Sep 11 18:27:50.503932 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:50.503571 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e9b9a66-30ad-4d94-95aa-d09ee8eea072-istio-data\") pod \"e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd\" (UID: \"7e9b9a66-30ad-4d94-95aa-d09ee8eea072\") " pod="openshift-ingress/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd" Sep 11 18:27:50.503932 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:50.503682 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e9b9a66-30ad-4d94-95aa-d09ee8eea072-workload-certs\") pod \"e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd\" (UID: \"7e9b9a66-30ad-4d94-95aa-d09ee8eea072\") " pod="openshift-ingress/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd" Sep 11 18:27:50.504979 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:50.504956 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e9b9a66-30ad-4d94-95aa-d09ee8eea072-istiod-ca-cert\") pod \"e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd\" (UID: \"7e9b9a66-30ad-4d94-95aa-d09ee8eea072\") " pod="openshift-ingress/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd" Sep 11 18:27:50.511474 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:50.511421 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e9b9a66-30ad-4d94-95aa-d09ee8eea072-istio-podinfo\") pod \"e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd\" (UID: \"7e9b9a66-30ad-4d94-95aa-d09ee8eea072\") " pod="openshift-ingress/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd" Sep 11 18:27:50.514084 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:50.514042 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5kg5\" (UniqueName: \"kubernetes.io/projected/7e9b9a66-30ad-4d94-95aa-d09ee8eea072-kube-api-access-k5kg5\") pod \"e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd\" (UID: \"7e9b9a66-30ad-4d94-95aa-d09ee8eea072\") " pod="openshift-ingress/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd" Sep 11 18:27:50.515136 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:50.514691 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e9b9a66-30ad-4d94-95aa-d09ee8eea072-istio-token\") pod \"e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd\" (UID: \"7e9b9a66-30ad-4d94-95aa-d09ee8eea072\") " pod="openshift-ingress/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd" Sep 11 18:27:50.515136 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:50.515079 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e9b9a66-30ad-4d94-95aa-d09ee8eea072-istio-envoy\") pod \"e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd\" (UID: \"7e9b9a66-30ad-4d94-95aa-d09ee8eea072\") " pod="openshift-ingress/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd" Sep 11 18:27:50.637566 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:50.637427 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd" Sep 11 18:27:50.954361 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:50.954319 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd"] Sep 11 18:27:50.956148 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:27:50.956118 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e9b9a66_30ad_4d94_95aa_d09ee8eea072.slice/crio-00813ffb68883db053e0e8c98fb49397c7f439c4e093747ad3b7c6fbac8fa3be WatchSource:0}: Error finding container 00813ffb68883db053e0e8c98fb49397c7f439c4e093747ad3b7c6fbac8fa3be: Status 404 returned error can't find the container with id 00813ffb68883db053e0e8c98fb49397c7f439c4e093747ad3b7c6fbac8fa3be Sep 11 18:27:50.959370 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:50.959332 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:27:50.959449 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:50.959402 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:27:50.959449 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:50.959434 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:27:51.071942 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:51.071594 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd" event={"ID":"7e9b9a66-30ad-4d94-95aa-d09ee8eea072","Type":"ContainerStarted","Data":"d695c9409d060d75a4b2891eb4a3090da2853e05ec90a20216eaaa7c45046e8c"} Sep 11 18:27:51.071942 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:51.071705 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd" event={"ID":"7e9b9a66-30ad-4d94-95aa-d09ee8eea072","Type":"ContainerStarted","Data":"00813ffb68883db053e0e8c98fb49397c7f439c4e093747ad3b7c6fbac8fa3be"} Sep 11 18:27:51.093283 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:51.093146 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd" podStartSLOduration=1.093126897 podStartE2EDuration="1.093126897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:27:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:27:51.090161993 +0000 UTC m=+752.587179750" watchObservedRunningTime="2026-09-11 18:27:51.093126897 +0000 UTC m=+752.590144652" Sep 11 18:27:51.637719 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:51.637679 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd" Sep 11 18:27:52.644504 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:52.644459 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd" Sep 11 18:27:53.091764 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:53.091726 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd" Sep 11 18:27:53.092756 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:53.092719 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd" Sep 11 18:27:54.098815 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:54.098780 2561 generic.go:356] "Generic (PLEG): container finished" podID="891f8576-360b-4004-b612-2169d1323649" containerID="7f53dc95d5b8ed078e3362f1f885c8b4f0ba38761ee23614e8b82934138015f7" exitCode=0 Sep 11 18:27:54.100450 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:54.098853 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm" event={"ID":"891f8576-360b-4004-b612-2169d1323649","Type":"ContainerDied","Data":"7f53dc95d5b8ed078e3362f1f885c8b4f0ba38761ee23614e8b82934138015f7"} Sep 11 18:27:54.100450 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:54.099326 2561 scope.go:117] "RemoveContainer" containerID="7f53dc95d5b8ed078e3362f1f885c8b4f0ba38761ee23614e8b82934138015f7" Sep 11 18:27:54.100858 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:54.100615 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:27:54.100858 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:54.100689 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:27:54.100858 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:54.100722 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:27:55.106865 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:55.106828 2561 generic.go:356] "Generic (PLEG): container finished" podID="11283c37-ec93-4ff1-a82b-ecc812c41b3d" containerID="4d7d1845627fb9b2afcf35ce69e88bae290d6c42957bb9c7184ae3f53c2bf247" exitCode=0 Sep 11 18:27:55.107363 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:55.106905 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s" event={"ID":"11283c37-ec93-4ff1-a82b-ecc812c41b3d","Type":"ContainerDied","Data":"4d7d1845627fb9b2afcf35ce69e88bae290d6c42957bb9c7184ae3f53c2bf247"} Sep 11 18:27:55.107460 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:55.107441 2561 scope.go:117] "RemoveContainer" containerID="4d7d1845627fb9b2afcf35ce69e88bae290d6c42957bb9c7184ae3f53c2bf247" Sep 11 18:27:55.108812 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:55.108781 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:27:55.108947 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:55.108857 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:27:55.108947 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:55.108904 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:27:55.109177 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:55.109159 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm" event={"ID":"891f8576-360b-4004-b612-2169d1323649","Type":"ContainerStarted","Data":"9db63a3867e2f67cc94e3282540a39d880f3cdca10d549c085bd50cceb730fdc"} Sep 11 18:27:55.480019 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:55.479980 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm" Sep 11 18:27:55.480311 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:55.480285 2561 patch_prober.go:28] interesting pod/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Sep 11 18:27:55.480376 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:55.480349 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm" podUID="891f8576-360b-4004-b612-2169d1323649" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Sep 11 18:27:56.124468 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:56.124421 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s" event={"ID":"11283c37-ec93-4ff1-a82b-ecc812c41b3d","Type":"ContainerStarted","Data":"ad1fbccb1aa2e54ac0091f14356d02bba2fd576f084bb858447f17deb4ead673"} Sep 11 18:27:56.412793 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:56.412755 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s" Sep 11 18:27:56.413204 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:56.413177 2561 patch_prober.go:28] interesting pod/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Sep 11 18:27:56.413420 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:56.413392 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s" podUID="11283c37-ec93-4ff1-a82b-ecc812c41b3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Sep 11 18:27:56.479953 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:56.479915 2561 patch_prober.go:28] interesting pod/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Sep 11 18:27:56.480153 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:56.479977 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm" podUID="891f8576-360b-4004-b612-2169d1323649" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Sep 11 18:27:57.413464 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:57.413423 2561 patch_prober.go:28] interesting pod/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Sep 11 18:27:57.414083 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:57.414015 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s" podUID="11283c37-ec93-4ff1-a82b-ecc812c41b3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Sep 11 18:27:57.480146 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:57.480102 2561 patch_prober.go:28] interesting pod/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Sep 11 18:27:57.480398 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:57.480361 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm" podUID="891f8576-360b-4004-b612-2169d1323649" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Sep 11 18:27:58.413664 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:58.413624 2561 patch_prober.go:28] interesting pod/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Sep 11 18:27:58.414273 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:58.414238 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s" podUID="11283c37-ec93-4ff1-a82b-ecc812c41b3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Sep 11 18:27:58.483680 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:58.483634 2561 patch_prober.go:28] interesting pod/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Sep 11 18:27:58.483983 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:58.483929 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm" podUID="891f8576-360b-4004-b612-2169d1323649" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Sep 11 18:27:59.417078 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:59.413402 2561 patch_prober.go:28] interesting pod/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Sep 11 18:27:59.417078 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:59.413468 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s" podUID="11283c37-ec93-4ff1-a82b-ecc812c41b3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Sep 11 18:27:59.480079 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:59.479987 2561 patch_prober.go:28] interesting pod/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Sep 11 18:27:59.480294 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:59.480080 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm" podUID="891f8576-360b-4004-b612-2169d1323649" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Sep 11 18:27:59.561384 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:59.561347 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1de19d-8659676f7d-q4qt4"] Sep 11 18:27:59.567834 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:59.565412 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1de19d-8659676f7d-q4qt4" Sep 11 18:27:59.574511 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:59.574479 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1de19d-8659676f7d-q4qt4"] Sep 11 18:27:59.580796 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:59.579659 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-1de19d\"" Sep 11 18:27:59.580796 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:59.579742 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-1de19d-dockercfg-bg8bv\"" Sep 11 18:27:59.656082 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:59.654199 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68ef7a27-900e-4d80-8c96-023cb2e14f21-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-1de19d-8659676f7d-q4qt4\" (UID: \"68ef7a27-900e-4d80-8c96-023cb2e14f21\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1de19d-8659676f7d-q4qt4" Sep 11 18:27:59.656082 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:59.654349 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk56b\" (UniqueName: \"kubernetes.io/projected/68ef7a27-900e-4d80-8c96-023cb2e14f21-kube-api-access-lk56b\") pod \"payload-pre-processing-e2e-inf-b-1de19d-8659676f7d-q4qt4\" (UID: \"68ef7a27-900e-4d80-8c96-023cb2e14f21\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1de19d-8659676f7d-q4qt4" Sep 11 18:27:59.778141 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:59.777862 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk56b\" (UniqueName: \"kubernetes.io/projected/68ef7a27-900e-4d80-8c96-023cb2e14f21-kube-api-access-lk56b\") pod \"payload-pre-processing-e2e-inf-b-1de19d-8659676f7d-q4qt4\" (UID: \"68ef7a27-900e-4d80-8c96-023cb2e14f21\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1de19d-8659676f7d-q4qt4" Sep 11 18:27:59.778141 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:59.778069 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68ef7a27-900e-4d80-8c96-023cb2e14f21-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-1de19d-8659676f7d-q4qt4\" (UID: \"68ef7a27-900e-4d80-8c96-023cb2e14f21\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1de19d-8659676f7d-q4qt4" Sep 11 18:27:59.782122 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:59.779159 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68ef7a27-900e-4d80-8c96-023cb2e14f21-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-1de19d-8659676f7d-q4qt4\" (UID: \"68ef7a27-900e-4d80-8c96-023cb2e14f21\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1de19d-8659676f7d-q4qt4" Sep 11 18:27:59.817833 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:59.817792 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk56b\" (UniqueName: \"kubernetes.io/projected/68ef7a27-900e-4d80-8c96-023cb2e14f21-kube-api-access-lk56b\") pod \"payload-pre-processing-e2e-inf-b-1de19d-8659676f7d-q4qt4\" (UID: \"68ef7a27-900e-4d80-8c96-023cb2e14f21\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1de19d-8659676f7d-q4qt4" Sep 11 18:27:59.896628 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:27:59.896281 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1de19d-8659676f7d-q4qt4" Sep 11 18:28:00.434080 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:00.434009 2561 patch_prober.go:28] interesting pod/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Sep 11 18:28:00.435033 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:00.434619 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s" podUID="11283c37-ec93-4ff1-a82b-ecc812c41b3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Sep 11 18:28:00.441087 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:00.438660 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1de19d-8659676f7d-q4qt4"] Sep 11 18:28:00.482493 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:00.482461 2561 patch_prober.go:28] interesting pod/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Sep 11 18:28:00.482938 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:00.482809 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm" podUID="891f8576-360b-4004-b612-2169d1323649" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Sep 11 18:28:01.188575 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:01.188537 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1de19d-8659676f7d-q4qt4" event={"ID":"68ef7a27-900e-4d80-8c96-023cb2e14f21","Type":"ContainerStarted","Data":"586bedfb21cc9e01eea07997973bb96589519a47403fb6ed8d20420e5d22d8f3"} Sep 11 18:28:01.188877 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:01.188852 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1de19d-8659676f7d-q4qt4" event={"ID":"68ef7a27-900e-4d80-8c96-023cb2e14f21","Type":"ContainerStarted","Data":"1ef353126260cb2f598d0ab247f4f4b0553939c3573aa99fbaef19fc5074faf0"} Sep 11 18:28:01.189916 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:01.189893 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1de19d-8659676f7d-q4qt4" Sep 11 18:28:01.208537 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:01.208467 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1de19d-8659676f7d-q4qt4" podStartSLOduration=2.20844375 podStartE2EDuration="2.20844375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:27:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:28:01.206182659 +0000 UTC m=+762.703200414" watchObservedRunningTime="2026-09-11 18:28:01.20844375 +0000 UTC m=+762.705461506" Sep 11 18:28:01.412760 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:01.412712 2561 patch_prober.go:28] interesting pod/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Sep 11 18:28:01.412970 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:01.412785 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s" podUID="11283c37-ec93-4ff1-a82b-ecc812c41b3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Sep 11 18:28:01.480707 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:01.480609 2561 patch_prober.go:28] interesting pod/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Sep 11 18:28:01.481297 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:01.481260 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm" podUID="891f8576-360b-4004-b612-2169d1323649" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Sep 11 18:28:02.417076 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:02.416585 2561 patch_prober.go:28] interesting pod/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Sep 11 18:28:02.417076 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:02.416689 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s" podUID="11283c37-ec93-4ff1-a82b-ecc812c41b3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Sep 11 18:28:02.480377 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:02.480128 2561 patch_prober.go:28] interesting pod/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Sep 11 18:28:02.480377 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:02.480199 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm" podUID="891f8576-360b-4004-b612-2169d1323649" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Sep 11 18:28:03.425226 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:03.425182 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s" Sep 11 18:28:03.427577 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:03.427544 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s" Sep 11 18:28:03.429031 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:03.429011 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s" Sep 11 18:28:03.494977 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:03.494938 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm" Sep 11 18:28:03.496647 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:03.496346 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm" Sep 11 18:28:03.498108 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:03.498083 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm" Sep 11 18:28:03.709148 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:03.709021 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8370956b-7js2v"] Sep 11 18:28:03.740927 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:03.740888 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8370956b-7js2v" Sep 11 18:28:03.742138 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:03.740748 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8370956b-7js2v"] Sep 11 18:28:03.744500 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:03.744467 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 18:28:03.744951 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:03.744922 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n968b\"" Sep 11 18:28:03.858629 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:03.858589 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18b886e3-dfd4-49c0-8fb1-1df83b55b990-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8370956b-7js2v\" (UID: \"18b886e3-dfd4-49c0-8fb1-1df83b55b990\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8370956b-7js2v" Sep 11 18:28:03.959813 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:03.959725 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18b886e3-dfd4-49c0-8fb1-1df83b55b990-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8370956b-7js2v\" (UID: \"18b886e3-dfd4-49c0-8fb1-1df83b55b990\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8370956b-7js2v" Sep 11 18:28:03.960463 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:03.960439 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18b886e3-dfd4-49c0-8fb1-1df83b55b990-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8370956b-7js2v\" (UID: \"18b886e3-dfd4-49c0-8fb1-1df83b55b990\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8370956b-7js2v" Sep 11 18:28:04.060072 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:04.060027 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8370956b-7js2v" Sep 11 18:28:04.393074 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:04.391391 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8370956b-7js2v"] Sep 11 18:28:05.235718 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:05.235685 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8370956b-7js2v" event={"ID":"18b886e3-dfd4-49c0-8fb1-1df83b55b990","Type":"ContainerStarted","Data":"46db14108c5707e603a2e3fc896f4a095d9bc5fd8e322116eb992eaeaa100543"} Sep 11 18:28:08.261790 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:08.261754 2561 generic.go:356] "Generic (PLEG): container finished" podID="18b886e3-dfd4-49c0-8fb1-1df83b55b990" containerID="f003b2e5e71f5b3e8bd3ab161125cd97a4a0996c2def74989639c624c703c8f1" exitCode=0 Sep 11 18:28:08.262251 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:08.261850 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8370956b-7js2v" event={"ID":"18b886e3-dfd4-49c0-8fb1-1df83b55b990","Type":"ContainerDied","Data":"f003b2e5e71f5b3e8bd3ab161125cd97a4a0996c2def74989639c624c703c8f1"} Sep 11 18:28:09.270002 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:09.269965 2561 generic.go:356] "Generic (PLEG): container finished" podID="90875473-2bb0-471b-97a4-467f6b10a468" containerID="466b8eae0deb060375948d4b461fb26118e3a59cd89efb40875656195860f1c2" exitCode=0 Sep 11 18:28:09.270564 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:09.270041 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c" event={"ID":"90875473-2bb0-471b-97a4-467f6b10a468","Type":"ContainerDied","Data":"466b8eae0deb060375948d4b461fb26118e3a59cd89efb40875656195860f1c2"} Sep 11 18:28:09.270655 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:09.270635 2561 scope.go:117] "RemoveContainer" containerID="466b8eae0deb060375948d4b461fb26118e3a59cd89efb40875656195860f1c2" Sep 11 18:28:09.271892 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:09.271868 2561 generic.go:356] "Generic (PLEG): container finished" podID="3730018e-ef6b-4210-9007-0861b85202df" containerID="966aca739243da692077f59bc34833a01cdffbc4013aa25842441a919628b33d" exitCode=0 Sep 11 18:28:09.272006 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:09.271930 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn" event={"ID":"3730018e-ef6b-4210-9007-0861b85202df","Type":"ContainerDied","Data":"966aca739243da692077f59bc34833a01cdffbc4013aa25842441a919628b33d"} Sep 11 18:28:09.272006 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:09.271921 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:28:09.272112 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:09.272093 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:28:09.272162 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:09.272147 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:28:09.272454 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:09.272438 2561 scope.go:117] "RemoveContainer" containerID="966aca739243da692077f59bc34833a01cdffbc4013aa25842441a919628b33d" Sep 11 18:28:09.273815 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:09.273785 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:28:09.273899 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:09.273864 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:28:09.273944 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:09.273915 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:28:09.476210 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:09.476185 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8370956b-7js2v" Sep 11 18:28:09.527830 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:09.527714 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18b886e3-dfd4-49c0-8fb1-1df83b55b990-maas-api-service-ca\") pod \"18b886e3-dfd4-49c0-8fb1-1df83b55b990\" (UID: \"18b886e3-dfd4-49c0-8fb1-1df83b55b990\") " Sep 11 18:28:09.528489 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:09.528452 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18b886e3-dfd4-49c0-8fb1-1df83b55b990-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "18b886e3-dfd4-49c0-8fb1-1df83b55b990" (UID: "18b886e3-dfd4-49c0-8fb1-1df83b55b990"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:28:09.629788 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:09.629550 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18b886e3-dfd4-49c0-8fb1-1df83b55b990-maas-api-service-ca\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:28:10.280146 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:10.280113 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8370956b-7js2v" Sep 11 18:28:10.280146 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:10.280130 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8370956b-7js2v" event={"ID":"18b886e3-dfd4-49c0-8fb1-1df83b55b990","Type":"ContainerDied","Data":"46db14108c5707e603a2e3fc896f4a095d9bc5fd8e322116eb992eaeaa100543"} Sep 11 18:28:10.280684 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:10.280183 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46db14108c5707e603a2e3fc896f4a095d9bc5fd8e322116eb992eaeaa100543" Sep 11 18:28:10.282601 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:10.282569 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c" event={"ID":"90875473-2bb0-471b-97a4-467f6b10a468","Type":"ContainerStarted","Data":"a081bf3097522808ffb227e403c5ac8580babe73e5794a777169215c82e35f0b"} Sep 11 18:28:10.285492 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:10.285462 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn" event={"ID":"3730018e-ef6b-4210-9007-0861b85202df","Type":"ContainerStarted","Data":"fd43669314cad07c05d63f7aa69d80d018c221a827b7edda4f013bfc919c1606"} Sep 11 18:28:10.766980 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:10.766939 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn" Sep 11 18:28:10.768276 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:10.768250 2561 patch_prober.go:28] interesting pod/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Sep 11 18:28:10.768405 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:10.768302 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn" podUID="3730018e-ef6b-4210-9007-0861b85202df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Sep 11 18:28:11.119806 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:11.119688 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c" Sep 11 18:28:11.121018 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:11.120980 2561 patch_prober.go:28] interesting pod/e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Sep 11 18:28:11.121175 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:11.121081 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c" podUID="90875473-2bb0-471b-97a4-467f6b10a468" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Sep 11 18:28:11.767571 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:11.767533 2561 patch_prober.go:28] interesting pod/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Sep 11 18:28:11.768044 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:11.768013 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn" podUID="3730018e-ef6b-4210-9007-0861b85202df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Sep 11 18:28:12.119553 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:12.119450 2561 patch_prober.go:28] interesting pod/e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Sep 11 18:28:12.119818 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:12.119787 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c" podUID="90875473-2bb0-471b-97a4-467f6b10a468" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Sep 11 18:28:12.766869 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:12.766836 2561 patch_prober.go:28] interesting pod/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Sep 11 18:28:12.767140 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:12.767111 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn" podUID="3730018e-ef6b-4210-9007-0861b85202df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Sep 11 18:28:12.986461 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:12.986419 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn" Sep 11 18:28:12.987898 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:12.987871 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c" Sep 11 18:28:13.119431 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:13.119330 2561 patch_prober.go:28] interesting pod/e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Sep 11 18:28:13.119715 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:13.119680 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c" podUID="90875473-2bb0-471b-97a4-467f6b10a468" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Sep 11 18:28:13.212302 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:13.212263 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1de19d-8659676f7d-q4qt4" Sep 11 18:28:13.767553 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:13.767513 2561 patch_prober.go:28] interesting pod/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Sep 11 18:28:13.767739 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:13.767570 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn" podUID="3730018e-ef6b-4210-9007-0861b85202df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Sep 11 18:28:14.120312 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:14.120206 2561 patch_prober.go:28] interesting pod/e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Sep 11 18:28:14.120312 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:14.120277 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c" podUID="90875473-2bb0-471b-97a4-467f6b10a468" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Sep 11 18:28:14.639277 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:14.639235 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8370956b-7js2v"] Sep 11 18:28:14.644553 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:14.644515 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8370956b-7js2v"] Sep 11 18:28:14.769660 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:14.769624 2561 patch_prober.go:28] interesting pod/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Sep 11 18:28:14.770100 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:14.770002 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn" podUID="3730018e-ef6b-4210-9007-0861b85202df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Sep 11 18:28:14.795193 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:14.795132 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c"] Sep 11 18:28:14.799674 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:14.799628 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c" podUID="90875473-2bb0-471b-97a4-467f6b10a468" containerName="istio-proxy" containerID="cri-o://a081bf3097522808ffb227e403c5ac8580babe73e5794a777169215c82e35f0b" gracePeriod=30 Sep 11 18:28:15.145991 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:15.145953 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18b886e3-dfd4-49c0-8fb1-1df83b55b990" path="/var/lib/kubelet/pods/18b886e3-dfd4-49c0-8fb1-1df83b55b990/volumes" Sep 11 18:28:15.767628 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:15.767576 2561 patch_prober.go:28] interesting pod/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Sep 11 18:28:15.767960 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:15.767907 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn" podUID="3730018e-ef6b-4210-9007-0861b85202df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Sep 11 18:28:16.775781 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:16.775726 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn" Sep 11 18:28:16.777886 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:16.777863 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn" Sep 11 18:28:21.676586 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:21.676557 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c" Sep 11 18:28:21.792217 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:21.792174 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90875473-2bb0-471b-97a4-467f6b10a468-istio-token\") pod \"90875473-2bb0-471b-97a4-467f6b10a468\" (UID: \"90875473-2bb0-471b-97a4-467f6b10a468\") " Sep 11 18:28:21.793467 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:21.792234 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nkhns\" (UniqueName: \"kubernetes.io/projected/90875473-2bb0-471b-97a4-467f6b10a468-kube-api-access-nkhns\") pod \"90875473-2bb0-471b-97a4-467f6b10a468\" (UID: \"90875473-2bb0-471b-97a4-467f6b10a468\") " Sep 11 18:28:21.793467 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:21.792353 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90875473-2bb0-471b-97a4-467f6b10a468-istio-envoy\") pod \"90875473-2bb0-471b-97a4-467f6b10a468\" (UID: \"90875473-2bb0-471b-97a4-467f6b10a468\") " Sep 11 18:28:21.793467 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:21.792408 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90875473-2bb0-471b-97a4-467f6b10a468-istio-podinfo\") pod \"90875473-2bb0-471b-97a4-467f6b10a468\" (UID: \"90875473-2bb0-471b-97a4-467f6b10a468\") " Sep 11 18:28:21.793467 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:21.792443 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90875473-2bb0-471b-97a4-467f6b10a468-istio-data\") pod \"90875473-2bb0-471b-97a4-467f6b10a468\" (UID: \"90875473-2bb0-471b-97a4-467f6b10a468\") " Sep 11 18:28:21.793467 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:21.792467 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90875473-2bb0-471b-97a4-467f6b10a468-workload-socket\") pod \"90875473-2bb0-471b-97a4-467f6b10a468\" (UID: \"90875473-2bb0-471b-97a4-467f6b10a468\") " Sep 11 18:28:21.793467 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:21.792579 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90875473-2bb0-471b-97a4-467f6b10a468-workload-certs\") pod \"90875473-2bb0-471b-97a4-467f6b10a468\" (UID: \"90875473-2bb0-471b-97a4-467f6b10a468\") " Sep 11 18:28:21.793467 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:21.792607 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90875473-2bb0-471b-97a4-467f6b10a468-istiod-ca-cert\") pod \"90875473-2bb0-471b-97a4-467f6b10a468\" (UID: \"90875473-2bb0-471b-97a4-467f6b10a468\") " Sep 11 18:28:21.793467 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:21.792638 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90875473-2bb0-471b-97a4-467f6b10a468-credential-socket\") pod \"90875473-2bb0-471b-97a4-467f6b10a468\" (UID: \"90875473-2bb0-471b-97a4-467f6b10a468\") " Sep 11 18:28:21.793467 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:21.792778 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90875473-2bb0-471b-97a4-467f6b10a468-istio-data" (OuterVolumeSpecName: "istio-data") pod "90875473-2bb0-471b-97a4-467f6b10a468" (UID: "90875473-2bb0-471b-97a4-467f6b10a468"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:28:21.793467 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:21.792880 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90875473-2bb0-471b-97a4-467f6b10a468-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "90875473-2bb0-471b-97a4-467f6b10a468" (UID: "90875473-2bb0-471b-97a4-467f6b10a468"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:28:21.793467 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:21.792894 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90875473-2bb0-471b-97a4-467f6b10a468-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "90875473-2bb0-471b-97a4-467f6b10a468" (UID: "90875473-2bb0-471b-97a4-467f6b10a468"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:28:21.793467 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:21.793070 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90875473-2bb0-471b-97a4-467f6b10a468-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "90875473-2bb0-471b-97a4-467f6b10a468" (UID: "90875473-2bb0-471b-97a4-467f6b10a468"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:28:21.793467 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:21.793151 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90875473-2bb0-471b-97a4-467f6b10a468-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "90875473-2bb0-471b-97a4-467f6b10a468" (UID: "90875473-2bb0-471b-97a4-467f6b10a468"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:28:21.793467 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:21.793212 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90875473-2bb0-471b-97a4-467f6b10a468-istio-data\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:28:21.793467 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:21.793232 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90875473-2bb0-471b-97a4-467f6b10a468-workload-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:28:21.793467 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:21.793248 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90875473-2bb0-471b-97a4-467f6b10a468-workload-certs\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:28:21.793467 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:21.793262 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90875473-2bb0-471b-97a4-467f6b10a468-credential-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:28:21.794958 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:21.794933 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90875473-2bb0-471b-97a4-467f6b10a468-istio-token" (OuterVolumeSpecName: "istio-token") pod "90875473-2bb0-471b-97a4-467f6b10a468" (UID: "90875473-2bb0-471b-97a4-467f6b10a468"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:28:21.795405 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:21.795380 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90875473-2bb0-471b-97a4-467f6b10a468-kube-api-access-nkhns" (OuterVolumeSpecName: "kube-api-access-nkhns") pod "90875473-2bb0-471b-97a4-467f6b10a468" (UID: "90875473-2bb0-471b-97a4-467f6b10a468"). InnerVolumeSpecName "kube-api-access-nkhns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:28:21.795405 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:21.795392 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90875473-2bb0-471b-97a4-467f6b10a468-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "90875473-2bb0-471b-97a4-467f6b10a468" (UID: "90875473-2bb0-471b-97a4-467f6b10a468"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:28:21.795584 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:21.795507 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90875473-2bb0-471b-97a4-467f6b10a468-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "90875473-2bb0-471b-97a4-467f6b10a468" (UID: "90875473-2bb0-471b-97a4-467f6b10a468"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 18:28:21.894391 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:21.894352 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90875473-2bb0-471b-97a4-467f6b10a468-istiod-ca-cert\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:28:21.894391 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:21.894385 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90875473-2bb0-471b-97a4-467f6b10a468-istio-token\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:28:21.894391 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:21.894398 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkhns\" (UniqueName: \"kubernetes.io/projected/90875473-2bb0-471b-97a4-467f6b10a468-kube-api-access-nkhns\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:28:21.894626 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:21.894409 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90875473-2bb0-471b-97a4-467f6b10a468-istio-envoy\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:28:21.894626 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:21.894417 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90875473-2bb0-471b-97a4-467f6b10a468-istio-podinfo\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:28:22.400337 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:22.400302 2561 generic.go:356] "Generic (PLEG): container finished" podID="90875473-2bb0-471b-97a4-467f6b10a468" containerID="a081bf3097522808ffb227e403c5ac8580babe73e5794a777169215c82e35f0b" exitCode=0 Sep 11 18:28:22.400549 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:22.400366 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c" Sep 11 18:28:22.400549 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:22.400383 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c" event={"ID":"90875473-2bb0-471b-97a4-467f6b10a468","Type":"ContainerDied","Data":"a081bf3097522808ffb227e403c5ac8580babe73e5794a777169215c82e35f0b"} Sep 11 18:28:22.400549 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:22.400446 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c" event={"ID":"90875473-2bb0-471b-97a4-467f6b10a468","Type":"ContainerDied","Data":"01a7dfd0ace3af1703b013bc17f28f68a7ae11cbd2c0a474c23f6f3069d05011"} Sep 11 18:28:22.400549 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:22.400469 2561 scope.go:117] "RemoveContainer" containerID="a081bf3097522808ffb227e403c5ac8580babe73e5794a777169215c82e35f0b" Sep 11 18:28:22.402415 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:22.402395 2561 generic.go:356] "Generic (PLEG): container finished" podID="7e9b9a66-30ad-4d94-95aa-d09ee8eea072" containerID="d695c9409d060d75a4b2891eb4a3090da2853e05ec90a20216eaaa7c45046e8c" exitCode=0 Sep 11 18:28:22.402533 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:22.402462 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd" event={"ID":"7e9b9a66-30ad-4d94-95aa-d09ee8eea072","Type":"ContainerDied","Data":"d695c9409d060d75a4b2891eb4a3090da2853e05ec90a20216eaaa7c45046e8c"} Sep 11 18:28:22.402827 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:22.402814 2561 scope.go:117] "RemoveContainer" containerID="d695c9409d060d75a4b2891eb4a3090da2853e05ec90a20216eaaa7c45046e8c" Sep 11 18:28:22.404251 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:22.404209 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:28:22.404355 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:22.404313 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:28:22.404400 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:22.404361 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:28:22.411840 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:22.411822 2561 scope.go:117] "RemoveContainer" containerID="466b8eae0deb060375948d4b461fb26118e3a59cd89efb40875656195860f1c2" Sep 11 18:28:22.421144 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:22.421120 2561 scope.go:117] "RemoveContainer" containerID="a081bf3097522808ffb227e403c5ac8580babe73e5794a777169215c82e35f0b" Sep 11 18:28:22.421455 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:28:22.421433 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a081bf3097522808ffb227e403c5ac8580babe73e5794a777169215c82e35f0b\": container with ID starting with a081bf3097522808ffb227e403c5ac8580babe73e5794a777169215c82e35f0b not found: ID does not exist" containerID="a081bf3097522808ffb227e403c5ac8580babe73e5794a777169215c82e35f0b" Sep 11 18:28:22.421513 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:22.421464 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a081bf3097522808ffb227e403c5ac8580babe73e5794a777169215c82e35f0b"} err="failed to get container status \"a081bf3097522808ffb227e403c5ac8580babe73e5794a777169215c82e35f0b\": rpc error: code = NotFound desc = could not find container \"a081bf3097522808ffb227e403c5ac8580babe73e5794a777169215c82e35f0b\": container with ID starting with a081bf3097522808ffb227e403c5ac8580babe73e5794a777169215c82e35f0b not found: ID does not exist" Sep 11 18:28:22.421513 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:22.421488 2561 scope.go:117] "RemoveContainer" containerID="466b8eae0deb060375948d4b461fb26118e3a59cd89efb40875656195860f1c2" Sep 11 18:28:22.421781 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:28:22.421758 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"466b8eae0deb060375948d4b461fb26118e3a59cd89efb40875656195860f1c2\": container with ID starting with 466b8eae0deb060375948d4b461fb26118e3a59cd89efb40875656195860f1c2 not found: ID does not exist" containerID="466b8eae0deb060375948d4b461fb26118e3a59cd89efb40875656195860f1c2" Sep 11 18:28:22.421862 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:22.421793 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"466b8eae0deb060375948d4b461fb26118e3a59cd89efb40875656195860f1c2"} err="failed to get container status \"466b8eae0deb060375948d4b461fb26118e3a59cd89efb40875656195860f1c2\": rpc error: code = NotFound desc = could not find container \"466b8eae0deb060375948d4b461fb26118e3a59cd89efb40875656195860f1c2\": container with ID starting with 466b8eae0deb060375948d4b461fb26118e3a59cd89efb40875656195860f1c2 not found: ID does not exist" Sep 11 18:28:22.444868 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:22.444835 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c"] Sep 11 18:28:22.451363 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:22.451333 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-8370956b-openshift-default-78b64f6756-k8p9c"] Sep 11 18:28:23.092521 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:23.092410 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd" Sep 11 18:28:23.139897 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:23.139866 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90875473-2bb0-471b-97a4-467f6b10a468" path="/var/lib/kubelet/pods/90875473-2bb0-471b-97a4-467f6b10a468/volumes" Sep 11 18:28:23.415938 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:23.415904 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd" event={"ID":"7e9b9a66-30ad-4d94-95aa-d09ee8eea072","Type":"ContainerStarted","Data":"041314b9ddd7eb2d28344155844f12a64cd53693deb0b1a532717b2252a1b598"} Sep 11 18:28:23.637913 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:23.637872 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd" Sep 11 18:28:23.639133 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:23.639108 2561 patch_prober.go:28] interesting pod/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Sep 11 18:28:23.639196 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:23.639174 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd" podUID="7e9b9a66-30ad-4d94-95aa-d09ee8eea072" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Sep 11 18:28:24.638298 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:24.638258 2561 patch_prober.go:28] interesting pod/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Sep 11 18:28:24.638684 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:24.638321 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd" podUID="7e9b9a66-30ad-4d94-95aa-d09ee8eea072" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Sep 11 18:28:25.638897 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:25.638855 2561 patch_prober.go:28] interesting pod/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Sep 11 18:28:25.639332 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:25.638951 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd" podUID="7e9b9a66-30ad-4d94-95aa-d09ee8eea072" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Sep 11 18:28:26.638487 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:26.638450 2561 patch_prober.go:28] interesting pod/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Sep 11 18:28:26.638712 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:26.638516 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd" podUID="7e9b9a66-30ad-4d94-95aa-d09ee8eea072" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Sep 11 18:28:27.638730 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:27.638690 2561 patch_prober.go:28] interesting pod/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Sep 11 18:28:27.639156 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:27.638776 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd" podUID="7e9b9a66-30ad-4d94-95aa-d09ee8eea072" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Sep 11 18:28:28.642529 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:28.642500 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd" Sep 11 18:28:28.642915 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:28.642877 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd" Sep 11 18:28:28.643529 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:28.643512 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd" Sep 11 18:28:43.083496 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:43.083411 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-839ba106-nzzw9"] Sep 11 18:28:43.084124 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:43.084105 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18b886e3-dfd4-49c0-8fb1-1df83b55b990" containerName="revoke-keys" Sep 11 18:28:43.084202 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:43.084127 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="18b886e3-dfd4-49c0-8fb1-1df83b55b990" containerName="revoke-keys" Sep 11 18:28:43.084202 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:43.084151 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90875473-2bb0-471b-97a4-467f6b10a468" containerName="istio-proxy" Sep 11 18:28:43.084202 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:43.084160 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="90875473-2bb0-471b-97a4-467f6b10a468" containerName="istio-proxy" Sep 11 18:28:43.084202 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:43.084172 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90875473-2bb0-471b-97a4-467f6b10a468" containerName="istio-proxy" Sep 11 18:28:43.084202 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:43.084182 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="90875473-2bb0-471b-97a4-467f6b10a468" containerName="istio-proxy" Sep 11 18:28:43.084370 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:43.084289 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="90875473-2bb0-471b-97a4-467f6b10a468" containerName="istio-proxy" Sep 11 18:28:43.084370 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:43.084300 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="18b886e3-dfd4-49c0-8fb1-1df83b55b990" containerName="revoke-keys" Sep 11 18:28:43.087345 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:43.087320 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-839ba106-nzzw9" Sep 11 18:28:43.090210 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:43.090184 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n968b\"" Sep 11 18:28:43.090540 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:43.090297 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 18:28:43.093845 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:43.093818 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-839ba106-nzzw9"] Sep 11 18:28:43.229310 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:43.229261 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97ea89a3-aa6e-454e-9815-eadaedefe139-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-839ba106-nzzw9\" (UID: \"97ea89a3-aa6e-454e-9815-eadaedefe139\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-839ba106-nzzw9" Sep 11 18:28:43.331181 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:43.331146 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97ea89a3-aa6e-454e-9815-eadaedefe139-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-839ba106-nzzw9\" (UID: \"97ea89a3-aa6e-454e-9815-eadaedefe139\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-839ba106-nzzw9" Sep 11 18:28:43.331802 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:43.331779 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97ea89a3-aa6e-454e-9815-eadaedefe139-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-839ba106-nzzw9\" (UID: \"97ea89a3-aa6e-454e-9815-eadaedefe139\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-839ba106-nzzw9" Sep 11 18:28:43.412249 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:43.412218 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-839ba106-nzzw9" Sep 11 18:28:43.688237 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:43.688005 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-839ba106-nzzw9"] Sep 11 18:28:43.688377 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:28:43.688280 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97ea89a3_aa6e_454e_9815_eadaedefe139.slice/crio-93b684ea977b6efbcc349d7d9ed1ed1c560cea20e82dcc14133d0a6444545353 WatchSource:0}: Error finding container 93b684ea977b6efbcc349d7d9ed1ed1c560cea20e82dcc14133d0a6444545353: Status 404 returned error can't find the container with id 93b684ea977b6efbcc349d7d9ed1ed1c560cea20e82dcc14133d0a6444545353 Sep 11 18:28:44.564367 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:44.564327 2561 generic.go:356] "Generic (PLEG): container finished" podID="97ea89a3-aa6e-454e-9815-eadaedefe139" containerID="c4dd24bbb5d36e1ecc1867c6fb2ba46ac67ede62b254a18eb6cb3086928ab0e8" exitCode=0 Sep 11 18:28:44.564783 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:44.564379 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-839ba106-nzzw9" event={"ID":"97ea89a3-aa6e-454e-9815-eadaedefe139","Type":"ContainerDied","Data":"c4dd24bbb5d36e1ecc1867c6fb2ba46ac67ede62b254a18eb6cb3086928ab0e8"} Sep 11 18:28:44.564783 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:44.564423 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-839ba106-nzzw9" event={"ID":"97ea89a3-aa6e-454e-9815-eadaedefe139","Type":"ContainerStarted","Data":"93b684ea977b6efbcc349d7d9ed1ed1c560cea20e82dcc14133d0a6444545353"} Sep 11 18:28:45.720696 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:45.720668 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-839ba106-nzzw9" Sep 11 18:28:45.857577 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:45.857470 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97ea89a3-aa6e-454e-9815-eadaedefe139-maas-api-service-ca\") pod \"97ea89a3-aa6e-454e-9815-eadaedefe139\" (UID: \"97ea89a3-aa6e-454e-9815-eadaedefe139\") " Sep 11 18:28:45.857873 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:45.857846 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97ea89a3-aa6e-454e-9815-eadaedefe139-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "97ea89a3-aa6e-454e-9815-eadaedefe139" (UID: "97ea89a3-aa6e-454e-9815-eadaedefe139"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:28:45.857980 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:45.857965 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97ea89a3-aa6e-454e-9815-eadaedefe139-maas-api-service-ca\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:28:46.578660 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:46.578631 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-839ba106-nzzw9" Sep 11 18:28:46.578832 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:46.578623 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-839ba106-nzzw9" event={"ID":"97ea89a3-aa6e-454e-9815-eadaedefe139","Type":"ContainerDied","Data":"93b684ea977b6efbcc349d7d9ed1ed1c560cea20e82dcc14133d0a6444545353"} Sep 11 18:28:46.578832 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:46.578763 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93b684ea977b6efbcc349d7d9ed1ed1c560cea20e82dcc14133d0a6444545353" Sep 11 18:28:49.854081 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:49.854029 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-839ba106-nzzw9"] Sep 11 18:28:49.857714 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:49.857685 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-839ba106-nzzw9"] Sep 11 18:28:50.711914 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:50.711874 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg"] Sep 11 18:28:50.712344 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:50.712329 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97ea89a3-aa6e-454e-9815-eadaedefe139" containerName="revoke-keys" Sep 11 18:28:50.712403 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:50.712345 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="97ea89a3-aa6e-454e-9815-eadaedefe139" containerName="revoke-keys" Sep 11 18:28:50.712863 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:50.712435 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="97ea89a3-aa6e-454e-9815-eadaedefe139" containerName="revoke-keys" Sep 11 18:28:50.712863 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:50.712445 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="90875473-2bb0-471b-97a4-467f6b10a468" containerName="istio-proxy" Sep 11 18:28:50.714634 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:50.714615 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg" Sep 11 18:28:50.717205 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:50.717181 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-3565479f-openshift-default-dockercfg-dspdl\"" Sep 11 18:28:50.728218 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:50.728185 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg"] Sep 11 18:28:50.806025 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:50.805987 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1156544b-531e-4078-baa5-a872554cdde7-workload-socket\") pod \"e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg\" (UID: \"1156544b-531e-4078-baa5-a872554cdde7\") " pod="openshift-ingress/e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg" Sep 11 18:28:50.806025 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:50.806030 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1156544b-531e-4078-baa5-a872554cdde7-istiod-ca-cert\") pod \"e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg\" (UID: \"1156544b-531e-4078-baa5-a872554cdde7\") " pod="openshift-ingress/e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg" Sep 11 18:28:50.806025 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:50.806073 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1156544b-531e-4078-baa5-a872554cdde7-istio-envoy\") pod \"e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg\" (UID: \"1156544b-531e-4078-baa5-a872554cdde7\") " pod="openshift-ingress/e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg" Sep 11 18:28:50.806308 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:50.806134 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1156544b-531e-4078-baa5-a872554cdde7-istio-podinfo\") pod \"e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg\" (UID: \"1156544b-531e-4078-baa5-a872554cdde7\") " pod="openshift-ingress/e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg" Sep 11 18:28:50.806308 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:50.806206 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqbx5\" (UniqueName: \"kubernetes.io/projected/1156544b-531e-4078-baa5-a872554cdde7-kube-api-access-jqbx5\") pod \"e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg\" (UID: \"1156544b-531e-4078-baa5-a872554cdde7\") " pod="openshift-ingress/e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg" Sep 11 18:28:50.806308 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:50.806232 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1156544b-531e-4078-baa5-a872554cdde7-istio-token\") pod \"e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg\" (UID: \"1156544b-531e-4078-baa5-a872554cdde7\") " pod="openshift-ingress/e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg" Sep 11 18:28:50.806308 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:50.806261 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1156544b-531e-4078-baa5-a872554cdde7-workload-certs\") pod \"e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg\" (UID: \"1156544b-531e-4078-baa5-a872554cdde7\") " pod="openshift-ingress/e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg" Sep 11 18:28:50.806308 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:50.806303 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1156544b-531e-4078-baa5-a872554cdde7-istio-data\") pod \"e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg\" (UID: \"1156544b-531e-4078-baa5-a872554cdde7\") " pod="openshift-ingress/e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg" Sep 11 18:28:50.806488 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:50.806349 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1156544b-531e-4078-baa5-a872554cdde7-credential-socket\") pod \"e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg\" (UID: \"1156544b-531e-4078-baa5-a872554cdde7\") " pod="openshift-ingress/e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg" Sep 11 18:28:50.907048 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:50.907015 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1156544b-531e-4078-baa5-a872554cdde7-workload-socket\") pod \"e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg\" (UID: \"1156544b-531e-4078-baa5-a872554cdde7\") " pod="openshift-ingress/e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg" Sep 11 18:28:50.907048 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:50.907073 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1156544b-531e-4078-baa5-a872554cdde7-istiod-ca-cert\") pod \"e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg\" (UID: \"1156544b-531e-4078-baa5-a872554cdde7\") " pod="openshift-ingress/e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg" Sep 11 18:28:50.907587 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:50.907103 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1156544b-531e-4078-baa5-a872554cdde7-istio-envoy\") pod \"e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg\" (UID: \"1156544b-531e-4078-baa5-a872554cdde7\") " pod="openshift-ingress/e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg" Sep 11 18:28:50.907587 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:50.907129 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1156544b-531e-4078-baa5-a872554cdde7-istio-podinfo\") pod \"e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg\" (UID: \"1156544b-531e-4078-baa5-a872554cdde7\") " pod="openshift-ingress/e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg" Sep 11 18:28:50.907587 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:50.907306 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqbx5\" (UniqueName: \"kubernetes.io/projected/1156544b-531e-4078-baa5-a872554cdde7-kube-api-access-jqbx5\") pod \"e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg\" (UID: \"1156544b-531e-4078-baa5-a872554cdde7\") " pod="openshift-ingress/e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg" Sep 11 18:28:50.907587 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:50.907367 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1156544b-531e-4078-baa5-a872554cdde7-istio-token\") pod \"e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg\" (UID: \"1156544b-531e-4078-baa5-a872554cdde7\") " pod="openshift-ingress/e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg" Sep 11 18:28:50.907587 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:50.907401 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1156544b-531e-4078-baa5-a872554cdde7-workload-certs\") pod \"e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg\" (UID: \"1156544b-531e-4078-baa5-a872554cdde7\") " pod="openshift-ingress/e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg" Sep 11 18:28:50.907587 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:50.907407 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1156544b-531e-4078-baa5-a872554cdde7-workload-socket\") pod \"e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg\" (UID: \"1156544b-531e-4078-baa5-a872554cdde7\") " pod="openshift-ingress/e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg" Sep 11 18:28:50.907587 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:50.907454 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1156544b-531e-4078-baa5-a872554cdde7-istio-data\") pod \"e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg\" (UID: \"1156544b-531e-4078-baa5-a872554cdde7\") " pod="openshift-ingress/e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg" Sep 11 18:28:50.907587 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:50.907521 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1156544b-531e-4078-baa5-a872554cdde7-credential-socket\") pod \"e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg\" (UID: \"1156544b-531e-4078-baa5-a872554cdde7\") " pod="openshift-ingress/e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg" Sep 11 18:28:50.907966 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:50.907752 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1156544b-531e-4078-baa5-a872554cdde7-workload-certs\") pod \"e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg\" (UID: \"1156544b-531e-4078-baa5-a872554cdde7\") " pod="openshift-ingress/e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg" Sep 11 18:28:50.907966 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:50.907827 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1156544b-531e-4078-baa5-a872554cdde7-credential-socket\") pod \"e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg\" (UID: \"1156544b-531e-4078-baa5-a872554cdde7\") " pod="openshift-ingress/e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg" Sep 11 18:28:50.907966 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:50.907888 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1156544b-531e-4078-baa5-a872554cdde7-istiod-ca-cert\") pod \"e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg\" (UID: \"1156544b-531e-4078-baa5-a872554cdde7\") " pod="openshift-ingress/e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg" Sep 11 18:28:50.907966 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:50.907950 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1156544b-531e-4078-baa5-a872554cdde7-istio-data\") pod \"e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg\" (UID: \"1156544b-531e-4078-baa5-a872554cdde7\") " pod="openshift-ingress/e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg" Sep 11 18:28:50.909394 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:50.909374 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1156544b-531e-4078-baa5-a872554cdde7-istio-envoy\") pod \"e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg\" (UID: \"1156544b-531e-4078-baa5-a872554cdde7\") " pod="openshift-ingress/e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg" Sep 11 18:28:50.909707 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:50.909690 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1156544b-531e-4078-baa5-a872554cdde7-istio-podinfo\") pod \"e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg\" (UID: \"1156544b-531e-4078-baa5-a872554cdde7\") " pod="openshift-ingress/e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg" Sep 11 18:28:50.917099 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:50.917028 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1156544b-531e-4078-baa5-a872554cdde7-istio-token\") pod \"e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg\" (UID: \"1156544b-531e-4078-baa5-a872554cdde7\") " pod="openshift-ingress/e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg" Sep 11 18:28:50.917246 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:50.917191 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqbx5\" (UniqueName: \"kubernetes.io/projected/1156544b-531e-4078-baa5-a872554cdde7-kube-api-access-jqbx5\") pod \"e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg\" (UID: \"1156544b-531e-4078-baa5-a872554cdde7\") " pod="openshift-ingress/e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg" Sep 11 18:28:51.027985 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:51.027878 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg" Sep 11 18:28:51.148227 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:51.148190 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97ea89a3-aa6e-454e-9815-eadaedefe139" path="/var/lib/kubelet/pods/97ea89a3-aa6e-454e-9815-eadaedefe139/volumes" Sep 11 18:28:51.313868 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:51.313833 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg"] Sep 11 18:28:51.315602 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:28:51.315555 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1156544b_531e_4078_baa5_a872554cdde7.slice/crio-72382bb47b8cfae8d3a1fb7d0c0a7f53df4149bb845479095d70d6087b082595 WatchSource:0}: Error finding container 72382bb47b8cfae8d3a1fb7d0c0a7f53df4149bb845479095d70d6087b082595: Status 404 returned error can't find the container with id 72382bb47b8cfae8d3a1fb7d0c0a7f53df4149bb845479095d70d6087b082595 Sep 11 18:28:51.318273 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:51.318236 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:28:51.318368 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:51.318311 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:28:51.318368 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:51.318342 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:28:51.613162 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:51.613074 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg" event={"ID":"1156544b-531e-4078-baa5-a872554cdde7","Type":"ContainerStarted","Data":"15ab9ce0a20cf85dd46b73db9445fca0da9c35d9d88302391fc83b0948b70134"} Sep 11 18:28:51.613162 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:51.613125 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg" event={"ID":"1156544b-531e-4078-baa5-a872554cdde7","Type":"ContainerStarted","Data":"72382bb47b8cfae8d3a1fb7d0c0a7f53df4149bb845479095d70d6087b082595"} Sep 11 18:28:51.634092 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:51.634018 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg" podStartSLOduration=1.6340027 podStartE2EDuration="1.6340027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:28:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:28:51.631014589 +0000 UTC m=+813.128032344" watchObservedRunningTime="2026-09-11 18:28:51.6340027 +0000 UTC m=+813.131020453" Sep 11 18:28:52.028392 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:52.028352 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg" Sep 11 18:28:52.033834 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:52.033805 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg" Sep 11 18:28:52.619998 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:52.619967 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg" Sep 11 18:28:52.621120 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:28:52.621091 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg" Sep 11 18:29:14.773942 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:14.773906 2561 generic.go:356] "Generic (PLEG): container finished" podID="1156544b-531e-4078-baa5-a872554cdde7" containerID="15ab9ce0a20cf85dd46b73db9445fca0da9c35d9d88302391fc83b0948b70134" exitCode=0 Sep 11 18:29:14.774380 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:14.773989 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg" event={"ID":"1156544b-531e-4078-baa5-a872554cdde7","Type":"ContainerDied","Data":"15ab9ce0a20cf85dd46b73db9445fca0da9c35d9d88302391fc83b0948b70134"} Sep 11 18:29:14.774522 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:14.774504 2561 scope.go:117] "RemoveContainer" containerID="15ab9ce0a20cf85dd46b73db9445fca0da9c35d9d88302391fc83b0948b70134" Sep 11 18:29:14.775851 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:14.775818 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:29:14.775963 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:14.775905 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:29:14.775963 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:14.775951 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:29:15.786539 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:15.786479 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg" event={"ID":"1156544b-531e-4078-baa5-a872554cdde7","Type":"ContainerStarted","Data":"63fec5ba69eeb1882a12d071c70c222df3464aecd4c0e9f2763da7613771ecd3"} Sep 11 18:29:16.028393 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:16.028353 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg" Sep 11 18:29:16.028606 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:16.028552 2561 patch_prober.go:28] interesting pod/e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Sep 11 18:29:16.028664 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:16.028603 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg" podUID="1156544b-531e-4078-baa5-a872554cdde7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Sep 11 18:29:17.029271 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:17.029233 2561 patch_prober.go:28] interesting pod/e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Sep 11 18:29:17.029779 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:17.029303 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg" podUID="1156544b-531e-4078-baa5-a872554cdde7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Sep 11 18:29:18.028732 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:18.028690 2561 patch_prober.go:28] interesting pod/e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Sep 11 18:29:18.028915 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:18.028756 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg" podUID="1156544b-531e-4078-baa5-a872554cdde7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Sep 11 18:29:18.880416 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:18.880378 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3565479f-7kr5k"] Sep 11 18:29:18.882810 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:18.882785 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3565479f-7kr5k" Sep 11 18:29:18.887198 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:18.885292 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n968b\"" Sep 11 18:29:18.887198 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:18.885603 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 18:29:18.892696 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:18.892667 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3565479f-7kr5k"] Sep 11 18:29:19.023702 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:19.023655 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a75c2262-e46e-4fb8-9700-d68f258679fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3565479f-7kr5k\" (UID: \"a75c2262-e46e-4fb8-9700-d68f258679fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3565479f-7kr5k" Sep 11 18:29:19.028713 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:19.028681 2561 patch_prober.go:28] interesting pod/e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Sep 11 18:29:19.028870 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:19.028744 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg" podUID="1156544b-531e-4078-baa5-a872554cdde7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Sep 11 18:29:19.124940 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:19.124896 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a75c2262-e46e-4fb8-9700-d68f258679fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3565479f-7kr5k\" (UID: \"a75c2262-e46e-4fb8-9700-d68f258679fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3565479f-7kr5k" Sep 11 18:29:19.125618 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:19.125596 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a75c2262-e46e-4fb8-9700-d68f258679fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3565479f-7kr5k\" (UID: \"a75c2262-e46e-4fb8-9700-d68f258679fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3565479f-7kr5k" Sep 11 18:29:19.203728 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:19.203691 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3565479f-7kr5k" Sep 11 18:29:19.481018 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:19.480958 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3565479f-7kr5k"] Sep 11 18:29:19.826334 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:19.826245 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3565479f-7kr5k" event={"ID":"a75c2262-e46e-4fb8-9700-d68f258679fd","Type":"ContainerStarted","Data":"eeb06ad4135695bcc5810057c5bc4f56504a4c07c4c31f08c1927911312390fc"} Sep 11 18:29:20.029232 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:20.029193 2561 patch_prober.go:28] interesting pod/e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Sep 11 18:29:20.029730 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:20.029256 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg" podUID="1156544b-531e-4078-baa5-a872554cdde7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Sep 11 18:29:20.839843 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:20.839801 2561 generic.go:356] "Generic (PLEG): container finished" podID="a75c2262-e46e-4fb8-9700-d68f258679fd" containerID="41bb20498fddb7bcd523d9c4d4d9d2e1f8ef58af4dc22726f2b626a4389b0052" exitCode=0 Sep 11 18:29:20.840212 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:20.839920 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3565479f-7kr5k" event={"ID":"a75c2262-e46e-4fb8-9700-d68f258679fd","Type":"ContainerDied","Data":"41bb20498fddb7bcd523d9c4d4d9d2e1f8ef58af4dc22726f2b626a4389b0052"} Sep 11 18:29:21.036264 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:21.036227 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg" Sep 11 18:29:21.036704 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:21.036638 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg" Sep 11 18:29:21.037715 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:21.037694 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg" Sep 11 18:29:21.987587 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:21.987561 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3565479f-7kr5k" Sep 11 18:29:22.055221 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:22.055184 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a75c2262-e46e-4fb8-9700-d68f258679fd-maas-api-service-ca\") pod \"a75c2262-e46e-4fb8-9700-d68f258679fd\" (UID: \"a75c2262-e46e-4fb8-9700-d68f258679fd\") " Sep 11 18:29:22.055593 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:22.055522 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a75c2262-e46e-4fb8-9700-d68f258679fd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a75c2262-e46e-4fb8-9700-d68f258679fd" (UID: "a75c2262-e46e-4fb8-9700-d68f258679fd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:29:22.156938 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:22.156906 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a75c2262-e46e-4fb8-9700-d68f258679fd-maas-api-service-ca\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:29:22.853667 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:22.853628 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3565479f-7kr5k" event={"ID":"a75c2262-e46e-4fb8-9700-d68f258679fd","Type":"ContainerDied","Data":"eeb06ad4135695bcc5810057c5bc4f56504a4c07c4c31f08c1927911312390fc"} Sep 11 18:29:22.853836 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:22.853684 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eeb06ad4135695bcc5810057c5bc4f56504a4c07c4c31f08c1927911312390fc" Sep 11 18:29:22.853836 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:22.853648 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3565479f-7kr5k" Sep 11 18:29:25.489496 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:25.489459 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3565479f-7kr5k"] Sep 11 18:29:25.493336 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:25.493305 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3565479f-7kr5k"] Sep 11 18:29:26.609449 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:26.609416 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg"] Sep 11 18:29:26.610157 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:26.609655 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg" podUID="1156544b-531e-4078-baa5-a872554cdde7" containerName="istio-proxy" containerID="cri-o://63fec5ba69eeb1882a12d071c70c222df3464aecd4c0e9f2763da7613771ecd3" gracePeriod=30 Sep 11 18:29:27.139994 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:27.139958 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a75c2262-e46e-4fb8-9700-d68f258679fd" path="/var/lib/kubelet/pods/a75c2262-e46e-4fb8-9700-d68f258679fd/volumes" Sep 11 18:29:27.354688 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:27.354658 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc"] Sep 11 18:29:27.355142 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:27.355119 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a75c2262-e46e-4fb8-9700-d68f258679fd" containerName="revoke-keys" Sep 11 18:29:27.355142 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:27.355137 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a75c2262-e46e-4fb8-9700-d68f258679fd" containerName="revoke-keys" Sep 11 18:29:27.355342 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:27.355223 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a75c2262-e46e-4fb8-9700-d68f258679fd" containerName="revoke-keys" Sep 11 18:29:27.358643 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:27.358614 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc" Sep 11 18:29:27.361094 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:27.361042 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-68ac8212-openshift-default-dockercfg-ngfcj\"" Sep 11 18:29:27.372213 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:27.372189 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc"] Sep 11 18:29:27.412472 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:27.412438 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77efe2e5-edb2-484f-98cb-9543b6d630d7-istiod-ca-cert\") pod \"e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc\" (UID: \"77efe2e5-edb2-484f-98cb-9543b6d630d7\") " pod="openshift-ingress/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc" Sep 11 18:29:27.412639 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:27.412479 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77efe2e5-edb2-484f-98cb-9543b6d630d7-workload-certs\") pod \"e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc\" (UID: \"77efe2e5-edb2-484f-98cb-9543b6d630d7\") " pod="openshift-ingress/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc" Sep 11 18:29:27.412639 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:27.412513 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77efe2e5-edb2-484f-98cb-9543b6d630d7-workload-socket\") pod \"e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc\" (UID: \"77efe2e5-edb2-484f-98cb-9543b6d630d7\") " pod="openshift-ingress/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc" Sep 11 18:29:27.412639 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:27.412535 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77efe2e5-edb2-484f-98cb-9543b6d630d7-istio-token\") pod \"e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc\" (UID: \"77efe2e5-edb2-484f-98cb-9543b6d630d7\") " pod="openshift-ingress/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc" Sep 11 18:29:27.412639 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:27.412576 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77efe2e5-edb2-484f-98cb-9543b6d630d7-credential-socket\") pod \"e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc\" (UID: \"77efe2e5-edb2-484f-98cb-9543b6d630d7\") " pod="openshift-ingress/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc" Sep 11 18:29:27.412639 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:27.412633 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkrpl\" (UniqueName: \"kubernetes.io/projected/77efe2e5-edb2-484f-98cb-9543b6d630d7-kube-api-access-rkrpl\") pod \"e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc\" (UID: \"77efe2e5-edb2-484f-98cb-9543b6d630d7\") " pod="openshift-ingress/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc" Sep 11 18:29:27.412811 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:27.412654 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77efe2e5-edb2-484f-98cb-9543b6d630d7-istio-envoy\") pod \"e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc\" (UID: \"77efe2e5-edb2-484f-98cb-9543b6d630d7\") " pod="openshift-ingress/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc" Sep 11 18:29:27.412811 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:27.412679 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77efe2e5-edb2-484f-98cb-9543b6d630d7-istio-podinfo\") pod \"e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc\" (UID: \"77efe2e5-edb2-484f-98cb-9543b6d630d7\") " pod="openshift-ingress/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc" Sep 11 18:29:27.412811 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:27.412700 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77efe2e5-edb2-484f-98cb-9543b6d630d7-istio-data\") pod \"e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc\" (UID: \"77efe2e5-edb2-484f-98cb-9543b6d630d7\") " pod="openshift-ingress/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc" Sep 11 18:29:27.514321 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:27.514284 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77efe2e5-edb2-484f-98cb-9543b6d630d7-istio-token\") pod \"e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc\" (UID: \"77efe2e5-edb2-484f-98cb-9543b6d630d7\") " pod="openshift-ingress/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc" Sep 11 18:29:27.514505 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:27.514343 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77efe2e5-edb2-484f-98cb-9543b6d630d7-credential-socket\") pod \"e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc\" (UID: \"77efe2e5-edb2-484f-98cb-9543b6d630d7\") " pod="openshift-ingress/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc" Sep 11 18:29:27.514505 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:27.514408 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkrpl\" (UniqueName: \"kubernetes.io/projected/77efe2e5-edb2-484f-98cb-9543b6d630d7-kube-api-access-rkrpl\") pod \"e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc\" (UID: \"77efe2e5-edb2-484f-98cb-9543b6d630d7\") " pod="openshift-ingress/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc" Sep 11 18:29:27.514505 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:27.514439 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77efe2e5-edb2-484f-98cb-9543b6d630d7-istio-envoy\") pod \"e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc\" (UID: \"77efe2e5-edb2-484f-98cb-9543b6d630d7\") " pod="openshift-ingress/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc" Sep 11 18:29:27.514505 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:27.514480 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77efe2e5-edb2-484f-98cb-9543b6d630d7-istio-podinfo\") pod \"e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc\" (UID: \"77efe2e5-edb2-484f-98cb-9543b6d630d7\") " pod="openshift-ingress/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc" Sep 11 18:29:27.514732 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:27.514516 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77efe2e5-edb2-484f-98cb-9543b6d630d7-istio-data\") pod \"e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc\" (UID: \"77efe2e5-edb2-484f-98cb-9543b6d630d7\") " pod="openshift-ingress/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc" Sep 11 18:29:27.514732 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:27.514638 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77efe2e5-edb2-484f-98cb-9543b6d630d7-istiod-ca-cert\") pod \"e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc\" (UID: \"77efe2e5-edb2-484f-98cb-9543b6d630d7\") " pod="openshift-ingress/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc" Sep 11 18:29:27.514732 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:27.514670 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77efe2e5-edb2-484f-98cb-9543b6d630d7-workload-certs\") pod \"e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc\" (UID: \"77efe2e5-edb2-484f-98cb-9543b6d630d7\") " pod="openshift-ingress/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc" Sep 11 18:29:27.514732 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:27.514712 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77efe2e5-edb2-484f-98cb-9543b6d630d7-workload-socket\") pod \"e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc\" (UID: \"77efe2e5-edb2-484f-98cb-9543b6d630d7\") " pod="openshift-ingress/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc" Sep 11 18:29:27.514936 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:27.514797 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77efe2e5-edb2-484f-98cb-9543b6d630d7-credential-socket\") pod \"e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc\" (UID: \"77efe2e5-edb2-484f-98cb-9543b6d630d7\") " pod="openshift-ingress/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc" Sep 11 18:29:27.515000 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:27.514954 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77efe2e5-edb2-484f-98cb-9543b6d630d7-istio-data\") pod \"e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc\" (UID: \"77efe2e5-edb2-484f-98cb-9543b6d630d7\") " pod="openshift-ingress/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc" Sep 11 18:29:27.515184 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:27.515148 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77efe2e5-edb2-484f-98cb-9543b6d630d7-workload-socket\") pod \"e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc\" (UID: \"77efe2e5-edb2-484f-98cb-9543b6d630d7\") " pod="openshift-ingress/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc" Sep 11 18:29:27.515286 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:27.515200 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77efe2e5-edb2-484f-98cb-9543b6d630d7-workload-certs\") pod \"e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc\" (UID: \"77efe2e5-edb2-484f-98cb-9543b6d630d7\") " pod="openshift-ingress/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc" Sep 11 18:29:27.515575 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:27.515507 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77efe2e5-edb2-484f-98cb-9543b6d630d7-istiod-ca-cert\") pod \"e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc\" (UID: \"77efe2e5-edb2-484f-98cb-9543b6d630d7\") " pod="openshift-ingress/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc" Sep 11 18:29:27.516730 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:27.516711 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77efe2e5-edb2-484f-98cb-9543b6d630d7-istio-envoy\") pod \"e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc\" (UID: \"77efe2e5-edb2-484f-98cb-9543b6d630d7\") " pod="openshift-ingress/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc" Sep 11 18:29:27.517153 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:27.517131 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77efe2e5-edb2-484f-98cb-9543b6d630d7-istio-podinfo\") pod \"e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc\" (UID: \"77efe2e5-edb2-484f-98cb-9543b6d630d7\") " pod="openshift-ingress/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc" Sep 11 18:29:27.527598 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:27.527566 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77efe2e5-edb2-484f-98cb-9543b6d630d7-istio-token\") pod \"e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc\" (UID: \"77efe2e5-edb2-484f-98cb-9543b6d630d7\") " pod="openshift-ingress/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc" Sep 11 18:29:27.527697 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:27.527681 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkrpl\" (UniqueName: \"kubernetes.io/projected/77efe2e5-edb2-484f-98cb-9543b6d630d7-kube-api-access-rkrpl\") pod \"e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc\" (UID: \"77efe2e5-edb2-484f-98cb-9543b6d630d7\") " pod="openshift-ingress/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc" Sep 11 18:29:27.672787 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:27.672700 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc" Sep 11 18:29:27.975351 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:27.975323 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc"] Sep 11 18:29:27.978474 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:29:27.978441 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77efe2e5_edb2_484f_98cb_9543b6d630d7.slice/crio-cd23267e739998b58e9a333e39716f4f85241990719d5207f97ddad5bb0a3d6a WatchSource:0}: Error finding container cd23267e739998b58e9a333e39716f4f85241990719d5207f97ddad5bb0a3d6a: Status 404 returned error can't find the container with id cd23267e739998b58e9a333e39716f4f85241990719d5207f97ddad5bb0a3d6a Sep 11 18:29:27.981369 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:27.981332 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:29:27.981469 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:27.981418 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:29:27.981469 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:27.981461 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:29:28.896292 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:28.896252 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc" event={"ID":"77efe2e5-edb2-484f-98cb-9543b6d630d7","Type":"ContainerStarted","Data":"f272535e50c5914ca2c847b649a4bc914f0d1dfd07ecbeb5eb755712d78b62c3"} Sep 11 18:29:28.896685 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:28.896309 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc" event={"ID":"77efe2e5-edb2-484f-98cb-9543b6d630d7","Type":"ContainerStarted","Data":"cd23267e739998b58e9a333e39716f4f85241990719d5207f97ddad5bb0a3d6a"} Sep 11 18:29:28.916542 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:28.916481 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc" podStartSLOduration=1.916467528 podStartE2EDuration="1.916467528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:29:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:29:28.914791253 +0000 UTC m=+850.411808998" watchObservedRunningTime="2026-09-11 18:29:28.916467528 +0000 UTC m=+850.413485281" Sep 11 18:29:29.673684 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:29.673639 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc" Sep 11 18:29:29.679197 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:29.679173 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc" Sep 11 18:29:29.903890 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:29.903854 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc" Sep 11 18:29:29.904737 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:29.904719 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc" Sep 11 18:29:31.883459 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:31.883385 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg" Sep 11 18:29:31.920304 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:31.920265 2561 generic.go:356] "Generic (PLEG): container finished" podID="1156544b-531e-4078-baa5-a872554cdde7" containerID="63fec5ba69eeb1882a12d071c70c222df3464aecd4c0e9f2763da7613771ecd3" exitCode=0 Sep 11 18:29:31.920466 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:31.920348 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg" Sep 11 18:29:31.920466 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:31.920359 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg" event={"ID":"1156544b-531e-4078-baa5-a872554cdde7","Type":"ContainerDied","Data":"63fec5ba69eeb1882a12d071c70c222df3464aecd4c0e9f2763da7613771ecd3"} Sep 11 18:29:31.920466 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:31.920426 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg" event={"ID":"1156544b-531e-4078-baa5-a872554cdde7","Type":"ContainerDied","Data":"72382bb47b8cfae8d3a1fb7d0c0a7f53df4149bb845479095d70d6087b082595"} Sep 11 18:29:31.920466 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:31.920444 2561 scope.go:117] "RemoveContainer" containerID="63fec5ba69eeb1882a12d071c70c222df3464aecd4c0e9f2763da7613771ecd3" Sep 11 18:29:31.932727 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:31.932707 2561 scope.go:117] "RemoveContainer" containerID="15ab9ce0a20cf85dd46b73db9445fca0da9c35d9d88302391fc83b0948b70134" Sep 11 18:29:31.942694 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:31.942672 2561 scope.go:117] "RemoveContainer" containerID="63fec5ba69eeb1882a12d071c70c222df3464aecd4c0e9f2763da7613771ecd3" Sep 11 18:29:31.942987 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:29:31.942965 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63fec5ba69eeb1882a12d071c70c222df3464aecd4c0e9f2763da7613771ecd3\": container with ID starting with 63fec5ba69eeb1882a12d071c70c222df3464aecd4c0e9f2763da7613771ecd3 not found: ID does not exist" containerID="63fec5ba69eeb1882a12d071c70c222df3464aecd4c0e9f2763da7613771ecd3" Sep 11 18:29:31.943102 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:31.943002 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63fec5ba69eeb1882a12d071c70c222df3464aecd4c0e9f2763da7613771ecd3"} err="failed to get container status \"63fec5ba69eeb1882a12d071c70c222df3464aecd4c0e9f2763da7613771ecd3\": rpc error: code = NotFound desc = could not find container \"63fec5ba69eeb1882a12d071c70c222df3464aecd4c0e9f2763da7613771ecd3\": container with ID starting with 63fec5ba69eeb1882a12d071c70c222df3464aecd4c0e9f2763da7613771ecd3 not found: ID does not exist" Sep 11 18:29:31.943102 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:31.943035 2561 scope.go:117] "RemoveContainer" containerID="15ab9ce0a20cf85dd46b73db9445fca0da9c35d9d88302391fc83b0948b70134" Sep 11 18:29:31.943410 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:29:31.943371 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15ab9ce0a20cf85dd46b73db9445fca0da9c35d9d88302391fc83b0948b70134\": container with ID starting with 15ab9ce0a20cf85dd46b73db9445fca0da9c35d9d88302391fc83b0948b70134 not found: ID does not exist" containerID="15ab9ce0a20cf85dd46b73db9445fca0da9c35d9d88302391fc83b0948b70134" Sep 11 18:29:31.943451 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:31.943421 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15ab9ce0a20cf85dd46b73db9445fca0da9c35d9d88302391fc83b0948b70134"} err="failed to get container status \"15ab9ce0a20cf85dd46b73db9445fca0da9c35d9d88302391fc83b0948b70134\": rpc error: code = NotFound desc = could not find container \"15ab9ce0a20cf85dd46b73db9445fca0da9c35d9d88302391fc83b0948b70134\": container with ID starting with 15ab9ce0a20cf85dd46b73db9445fca0da9c35d9d88302391fc83b0948b70134 not found: ID does not exist" Sep 11 18:29:31.964303 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:31.964270 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1156544b-531e-4078-baa5-a872554cdde7-credential-socket\") pod \"1156544b-531e-4078-baa5-a872554cdde7\" (UID: \"1156544b-531e-4078-baa5-a872554cdde7\") " Sep 11 18:29:31.964456 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:31.964343 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jqbx5\" (UniqueName: \"kubernetes.io/projected/1156544b-531e-4078-baa5-a872554cdde7-kube-api-access-jqbx5\") pod \"1156544b-531e-4078-baa5-a872554cdde7\" (UID: \"1156544b-531e-4078-baa5-a872554cdde7\") " Sep 11 18:29:31.964456 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:31.964405 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1156544b-531e-4078-baa5-a872554cdde7-workload-certs\") pod \"1156544b-531e-4078-baa5-a872554cdde7\" (UID: \"1156544b-531e-4078-baa5-a872554cdde7\") " Sep 11 18:29:31.964456 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:31.964450 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1156544b-531e-4078-baa5-a872554cdde7-istio-podinfo\") pod \"1156544b-531e-4078-baa5-a872554cdde7\" (UID: \"1156544b-531e-4078-baa5-a872554cdde7\") " Sep 11 18:29:31.964652 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:31.964524 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1156544b-531e-4078-baa5-a872554cdde7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1156544b-531e-4078-baa5-a872554cdde7" (UID: "1156544b-531e-4078-baa5-a872554cdde7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:29:31.964652 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:31.964562 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1156544b-531e-4078-baa5-a872554cdde7-istiod-ca-cert\") pod \"1156544b-531e-4078-baa5-a872554cdde7\" (UID: \"1156544b-531e-4078-baa5-a872554cdde7\") " Sep 11 18:29:31.964652 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:31.964588 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1156544b-531e-4078-baa5-a872554cdde7-istio-token\") pod \"1156544b-531e-4078-baa5-a872554cdde7\" (UID: \"1156544b-531e-4078-baa5-a872554cdde7\") " Sep 11 18:29:31.964652 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:31.964623 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1156544b-531e-4078-baa5-a872554cdde7-istio-data\") pod \"1156544b-531e-4078-baa5-a872554cdde7\" (UID: \"1156544b-531e-4078-baa5-a872554cdde7\") " Sep 11 18:29:31.964864 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:31.964665 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1156544b-531e-4078-baa5-a872554cdde7-workload-socket\") pod \"1156544b-531e-4078-baa5-a872554cdde7\" (UID: \"1156544b-531e-4078-baa5-a872554cdde7\") " Sep 11 18:29:31.964864 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:31.964695 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1156544b-531e-4078-baa5-a872554cdde7-istio-envoy\") pod \"1156544b-531e-4078-baa5-a872554cdde7\" (UID: \"1156544b-531e-4078-baa5-a872554cdde7\") " Sep 11 18:29:31.964864 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:31.964711 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1156544b-531e-4078-baa5-a872554cdde7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1156544b-531e-4078-baa5-a872554cdde7" (UID: "1156544b-531e-4078-baa5-a872554cdde7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:29:31.965003 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:31.964934 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1156544b-531e-4078-baa5-a872554cdde7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1156544b-531e-4078-baa5-a872554cdde7" (UID: "1156544b-531e-4078-baa5-a872554cdde7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:29:31.965529 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:31.965198 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1156544b-531e-4078-baa5-a872554cdde7-istio-data" (OuterVolumeSpecName: "istio-data") pod "1156544b-531e-4078-baa5-a872554cdde7" (UID: "1156544b-531e-4078-baa5-a872554cdde7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:29:31.965529 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:31.965210 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1156544b-531e-4078-baa5-a872554cdde7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1156544b-531e-4078-baa5-a872554cdde7" (UID: "1156544b-531e-4078-baa5-a872554cdde7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:29:31.965529 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:31.965476 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1156544b-531e-4078-baa5-a872554cdde7-workload-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:29:31.965806 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:31.965757 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1156544b-531e-4078-baa5-a872554cdde7-credential-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:29:31.965806 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:31.965779 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1156544b-531e-4078-baa5-a872554cdde7-workload-certs\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:29:31.965806 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:31.965794 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1156544b-531e-4078-baa5-a872554cdde7-istiod-ca-cert\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:29:31.965979 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:31.965808 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1156544b-531e-4078-baa5-a872554cdde7-istio-data\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:29:31.966795 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:31.966765 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1156544b-531e-4078-baa5-a872554cdde7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1156544b-531e-4078-baa5-a872554cdde7" (UID: "1156544b-531e-4078-baa5-a872554cdde7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 18:29:31.966924 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:31.966871 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1156544b-531e-4078-baa5-a872554cdde7-kube-api-access-jqbx5" (OuterVolumeSpecName: "kube-api-access-jqbx5") pod "1156544b-531e-4078-baa5-a872554cdde7" (UID: "1156544b-531e-4078-baa5-a872554cdde7"). InnerVolumeSpecName "kube-api-access-jqbx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:29:31.967163 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:31.967141 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1156544b-531e-4078-baa5-a872554cdde7-istio-token" (OuterVolumeSpecName: "istio-token") pod "1156544b-531e-4078-baa5-a872554cdde7" (UID: "1156544b-531e-4078-baa5-a872554cdde7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:29:31.967260 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:31.967234 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1156544b-531e-4078-baa5-a872554cdde7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1156544b-531e-4078-baa5-a872554cdde7" (UID: "1156544b-531e-4078-baa5-a872554cdde7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:29:32.067150 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:32.067114 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1156544b-531e-4078-baa5-a872554cdde7-istio-podinfo\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:29:32.067150 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:32.067142 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1156544b-531e-4078-baa5-a872554cdde7-istio-token\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:29:32.067150 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:32.067151 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1156544b-531e-4078-baa5-a872554cdde7-istio-envoy\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:29:32.067150 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:32.067159 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqbx5\" (UniqueName: \"kubernetes.io/projected/1156544b-531e-4078-baa5-a872554cdde7-kube-api-access-jqbx5\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:29:32.250694 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:32.250657 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg"] Sep 11 18:29:32.253494 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:32.253464 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-3565479f-openshift-default-76b557bd74-mhxzg"] Sep 11 18:29:33.140165 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:33.140133 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1156544b-531e-4078-baa5-a872554cdde7" path="/var/lib/kubelet/pods/1156544b-531e-4078-baa5-a872554cdde7/volumes" Sep 11 18:29:45.947219 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:45.947177 2561 patch_prober.go:28] interesting pod/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 11 18:29:45.947609 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:45.947249 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc" podUID="77efe2e5-edb2-484f-98cb-9543b6d630d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 11 18:29:48.045151 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:48.045113 2561 generic.go:356] "Generic (PLEG): container finished" podID="77efe2e5-edb2-484f-98cb-9543b6d630d7" containerID="f272535e50c5914ca2c847b649a4bc914f0d1dfd07ecbeb5eb755712d78b62c3" exitCode=0 Sep 11 18:29:48.045582 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:48.045188 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc" event={"ID":"77efe2e5-edb2-484f-98cb-9543b6d630d7","Type":"ContainerDied","Data":"f272535e50c5914ca2c847b649a4bc914f0d1dfd07ecbeb5eb755712d78b62c3"} Sep 11 18:29:48.045646 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:48.045629 2561 scope.go:117] "RemoveContainer" containerID="f272535e50c5914ca2c847b649a4bc914f0d1dfd07ecbeb5eb755712d78b62c3" Sep 11 18:29:48.046825 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:48.046797 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:29:48.046887 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:48.046864 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:29:48.046929 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:48.046893 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:29:49.054091 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:49.054038 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc" event={"ID":"77efe2e5-edb2-484f-98cb-9543b6d630d7","Type":"ContainerStarted","Data":"0ce0c34054fb290e7729520ac1232a1d32258986599e777540a232294a63edbd"} Sep 11 18:29:49.673469 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:49.673434 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc" Sep 11 18:29:49.673754 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:49.673731 2561 patch_prober.go:28] interesting pod/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Sep 11 18:29:49.673838 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:49.673819 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc" podUID="77efe2e5-edb2-484f-98cb-9543b6d630d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Sep 11 18:29:50.673080 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:50.673022 2561 patch_prober.go:28] interesting pod/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Sep 11 18:29:50.673561 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:50.673103 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc" podUID="77efe2e5-edb2-484f-98cb-9543b6d630d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Sep 11 18:29:51.673174 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:51.673134 2561 patch_prober.go:28] interesting pod/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Sep 11 18:29:51.673572 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:51.673212 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc" podUID="77efe2e5-edb2-484f-98cb-9543b6d630d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Sep 11 18:29:52.674163 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:52.674107 2561 patch_prober.go:28] interesting pod/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Sep 11 18:29:52.674544 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:52.674202 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc" podUID="77efe2e5-edb2-484f-98cb-9543b6d630d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Sep 11 18:29:53.673460 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:53.673417 2561 patch_prober.go:28] interesting pod/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Sep 11 18:29:53.673646 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:53.673493 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc" podUID="77efe2e5-edb2-484f-98cb-9543b6d630d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Sep 11 18:29:54.744109 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:54.744042 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc" Sep 11 18:29:54.744552 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:54.744516 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc" Sep 11 18:29:54.747011 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:54.746989 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc" Sep 11 18:29:55.117214 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:55.117133 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68ac8212-xk4g7"] Sep 11 18:29:55.117567 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:55.117554 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1156544b-531e-4078-baa5-a872554cdde7" containerName="istio-proxy" Sep 11 18:29:55.117616 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:55.117569 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1156544b-531e-4078-baa5-a872554cdde7" containerName="istio-proxy" Sep 11 18:29:55.117661 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:55.117653 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1156544b-531e-4078-baa5-a872554cdde7" containerName="istio-proxy" Sep 11 18:29:55.117694 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:55.117664 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1156544b-531e-4078-baa5-a872554cdde7" containerName="istio-proxy" Sep 11 18:29:55.120897 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:55.120880 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68ac8212-xk4g7" Sep 11 18:29:55.123223 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:55.123201 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 18:29:55.123340 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:55.123206 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n968b\"" Sep 11 18:29:55.127790 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:55.127765 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68ac8212-xk4g7"] Sep 11 18:29:55.217498 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:55.217458 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52396ffe-7122-4a5a-8e4e-cf3ea4e87675-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-68ac8212-xk4g7\" (UID: \"52396ffe-7122-4a5a-8e4e-cf3ea4e87675\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68ac8212-xk4g7" Sep 11 18:29:55.318372 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:55.318334 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52396ffe-7122-4a5a-8e4e-cf3ea4e87675-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-68ac8212-xk4g7\" (UID: \"52396ffe-7122-4a5a-8e4e-cf3ea4e87675\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68ac8212-xk4g7" Sep 11 18:29:55.319083 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:55.319025 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52396ffe-7122-4a5a-8e4e-cf3ea4e87675-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-68ac8212-xk4g7\" (UID: \"52396ffe-7122-4a5a-8e4e-cf3ea4e87675\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68ac8212-xk4g7" Sep 11 18:29:55.433592 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:55.433559 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68ac8212-xk4g7" Sep 11 18:29:55.706987 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:55.706959 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68ac8212-xk4g7"] Sep 11 18:29:55.708560 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:29:55.708532 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52396ffe_7122_4a5a_8e4e_cf3ea4e87675.slice/crio-298b94082166482ed1cf19387c09fa0f8173db1186978f0349001ea15f9e2c2a WatchSource:0}: Error finding container 298b94082166482ed1cf19387c09fa0f8173db1186978f0349001ea15f9e2c2a: Status 404 returned error can't find the container with id 298b94082166482ed1cf19387c09fa0f8173db1186978f0349001ea15f9e2c2a Sep 11 18:29:56.105282 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:56.105242 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68ac8212-xk4g7" event={"ID":"52396ffe-7122-4a5a-8e4e-cf3ea4e87675","Type":"ContainerStarted","Data":"33458db655c8b15fd1e155b310051e8fb5a491f7197438a843502cbe1e71f843"} Sep 11 18:29:56.105675 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:56.105302 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68ac8212-xk4g7" event={"ID":"52396ffe-7122-4a5a-8e4e-cf3ea4e87675","Type":"ContainerStarted","Data":"298b94082166482ed1cf19387c09fa0f8173db1186978f0349001ea15f9e2c2a"} Sep 11 18:29:56.121173 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:56.121120 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68ac8212-xk4g7" podStartSLOduration=1.121103513 podStartE2EDuration="1.121103513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:29:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:29:56.119985368 +0000 UTC m=+877.617003122" watchObservedRunningTime="2026-09-11 18:29:56.121103513 +0000 UTC m=+877.618121266" Sep 11 18:29:57.113068 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:57.113013 2561 generic.go:356] "Generic (PLEG): container finished" podID="52396ffe-7122-4a5a-8e4e-cf3ea4e87675" containerID="33458db655c8b15fd1e155b310051e8fb5a491f7197438a843502cbe1e71f843" exitCode=0 Sep 11 18:29:57.113471 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:57.113081 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68ac8212-xk4g7" event={"ID":"52396ffe-7122-4a5a-8e4e-cf3ea4e87675","Type":"ContainerDied","Data":"33458db655c8b15fd1e155b310051e8fb5a491f7197438a843502cbe1e71f843"} Sep 11 18:29:58.265008 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:58.264985 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68ac8212-xk4g7" Sep 11 18:29:58.352082 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:58.352021 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52396ffe-7122-4a5a-8e4e-cf3ea4e87675-maas-api-service-ca\") pod \"52396ffe-7122-4a5a-8e4e-cf3ea4e87675\" (UID: \"52396ffe-7122-4a5a-8e4e-cf3ea4e87675\") " Sep 11 18:29:58.352392 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:58.352365 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52396ffe-7122-4a5a-8e4e-cf3ea4e87675-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "52396ffe-7122-4a5a-8e4e-cf3ea4e87675" (UID: "52396ffe-7122-4a5a-8e4e-cf3ea4e87675"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:29:58.453431 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:58.453392 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52396ffe-7122-4a5a-8e4e-cf3ea4e87675-maas-api-service-ca\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:29:59.129182 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:59.129104 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68ac8212-xk4g7" Sep 11 18:29:59.129182 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:59.129127 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68ac8212-xk4g7" event={"ID":"52396ffe-7122-4a5a-8e4e-cf3ea4e87675","Type":"ContainerDied","Data":"298b94082166482ed1cf19387c09fa0f8173db1186978f0349001ea15f9e2c2a"} Sep 11 18:29:59.129182 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:29:59.129175 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="298b94082166482ed1cf19387c09fa0f8173db1186978f0349001ea15f9e2c2a" Sep 11 18:30:00.136531 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:00.136500 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819190-q9l4g"] Sep 11 18:30:00.136943 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:00.136929 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52396ffe-7122-4a5a-8e4e-cf3ea4e87675" containerName="revoke-keys" Sep 11 18:30:00.136983 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:00.136944 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="52396ffe-7122-4a5a-8e4e-cf3ea4e87675" containerName="revoke-keys" Sep 11 18:30:00.136983 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:00.136968 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1156544b-531e-4078-baa5-a872554cdde7" containerName="istio-proxy" Sep 11 18:30:00.136983 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:00.136974 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1156544b-531e-4078-baa5-a872554cdde7" containerName="istio-proxy" Sep 11 18:30:00.137100 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:00.137045 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="52396ffe-7122-4a5a-8e4e-cf3ea4e87675" containerName="revoke-keys" Sep 11 18:30:00.145135 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:00.141525 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819190-q9l4g" Sep 11 18:30:00.147479 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:00.147450 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n968b\"" Sep 11 18:30:00.150803 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:00.150778 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-efa8b9-29819190-vjtwg"] Sep 11 18:30:00.154427 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:00.154399 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-724327-29819190-dq289"] Sep 11 18:30:00.154589 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:00.154565 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-efa8b9-29819190-vjtwg" Sep 11 18:30:00.159844 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:00.158700 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-724327-29819190-dq289" Sep 11 18:30:00.159844 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:00.159614 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819190-q9l4g" Sep 11 18:30:00.160352 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:00.160327 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819190-q9l4g"] Sep 11 18:30:00.166894 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:00.166864 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-efa8b9-29819190-vjtwg"] Sep 11 18:30:00.175647 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:00.175615 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-724327-29819190-dq289"] Sep 11 18:30:00.176006 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:00.175984 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-efa8b9-29819190-vjtwg" Sep 11 18:30:00.178441 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:00.178420 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-7e8ab9-29819190-7wmxb"] Sep 11 18:30:00.185252 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:00.185229 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-7e8ab9-29819190-7wmxb" Sep 11 18:30:00.186747 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:00.186703 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-7e8ab9-29819190-7wmxb"] Sep 11 18:30:00.222360 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:00.222330 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-724327-29819190-dq289" Sep 11 18:30:00.230979 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:00.230742 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-7e8ab9-29819190-7wmxb" Sep 11 18:30:00.470804 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:00.470771 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-efa8b9-29819190-vjtwg"] Sep 11 18:30:00.472116 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:30:00.472092 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod079c554e_e599_4ba7_a259_0a5e8cafad30.slice/crio-e6454290dca5fbd15f43acc7fe0b9fa9a87401f91455f8f66a68f6ba1dfc9b9a WatchSource:0}: Error finding container e6454290dca5fbd15f43acc7fe0b9fa9a87401f91455f8f66a68f6ba1dfc9b9a: Status 404 returned error can't find the container with id e6454290dca5fbd15f43acc7fe0b9fa9a87401f91455f8f66a68f6ba1dfc9b9a Sep 11 18:30:00.474104 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:30:00.474076 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c688bfb_48db_4bc1_839a_ee5f08c2994b.slice/crio-de6e104c6520502c86292c737b00c916e58dcfd5812d150d0a53df0827a0b7d3 WatchSource:0}: Error finding container de6e104c6520502c86292c737b00c916e58dcfd5812d150d0a53df0827a0b7d3: Status 404 returned error can't find the container with id de6e104c6520502c86292c737b00c916e58dcfd5812d150d0a53df0827a0b7d3 Sep 11 18:30:00.474709 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:00.474683 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819190-q9l4g"] Sep 11 18:30:00.533100 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:00.533042 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-7e8ab9-29819190-7wmxb"] Sep 11 18:30:00.533970 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:30:00.533937 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66f53630_fd06_4546_aef0_b09d82db862b.slice/crio-1e3ee72610e50897dae036d68767f1585530fb1b033473e4fcd1087acf06ca95 WatchSource:0}: Error finding container 1e3ee72610e50897dae036d68767f1585530fb1b033473e4fcd1087acf06ca95: Status 404 returned error can't find the container with id 1e3ee72610e50897dae036d68767f1585530fb1b033473e4fcd1087acf06ca95 Sep 11 18:30:00.534792 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:30:00.534764 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc23f483_c0dc_48d1_9a0c_5376be955b73.slice/crio-661ada728967ec4e2d7364790c5738f3b454dd37570f4a1d0456806588137f23 WatchSource:0}: Error finding container 661ada728967ec4e2d7364790c5738f3b454dd37570f4a1d0456806588137f23: Status 404 returned error can't find the container with id 661ada728967ec4e2d7364790c5738f3b454dd37570f4a1d0456806588137f23 Sep 11 18:30:00.536371 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:00.536349 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-724327-29819190-dq289"] Sep 11 18:30:01.144273 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:01.144172 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-7e8ab9-29819190-7wmxb" event={"ID":"66f53630-fd06-4546-aef0-b09d82db862b","Type":"ContainerStarted","Data":"a63bef399a98a4f6152d8ea5adae6399f6ddfd345155eec1acb8c4ef59bccf4e"} Sep 11 18:30:01.144273 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:01.144231 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-7e8ab9-29819190-7wmxb" event={"ID":"66f53630-fd06-4546-aef0-b09d82db862b","Type":"ContainerStarted","Data":"1e3ee72610e50897dae036d68767f1585530fb1b033473e4fcd1087acf06ca95"} Sep 11 18:30:01.145783 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:01.145755 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-efa8b9-29819190-vjtwg" event={"ID":"079c554e-e599-4ba7-a259-0a5e8cafad30","Type":"ContainerStarted","Data":"6de739ac64002a6ef60650d5bfedd1f00cc9efef65a305dae1c3c14d14d6ec0d"} Sep 11 18:30:01.145912 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:01.145812 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-efa8b9-29819190-vjtwg" event={"ID":"079c554e-e599-4ba7-a259-0a5e8cafad30","Type":"ContainerStarted","Data":"e6454290dca5fbd15f43acc7fe0b9fa9a87401f91455f8f66a68f6ba1dfc9b9a"} Sep 11 18:30:01.147459 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:01.147428 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819190-q9l4g" event={"ID":"6c688bfb-48db-4bc1-839a-ee5f08c2994b","Type":"ContainerStarted","Data":"f47f5dfdc0b1e8fa96894fb4c2e78ef85508065ae0fe98b18a7a7f84a1aa3ba6"} Sep 11 18:30:01.147601 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:01.147476 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819190-q9l4g" event={"ID":"6c688bfb-48db-4bc1-839a-ee5f08c2994b","Type":"ContainerStarted","Data":"de6e104c6520502c86292c737b00c916e58dcfd5812d150d0a53df0827a0b7d3"} Sep 11 18:30:01.148935 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:01.148912 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-724327-29819190-dq289" event={"ID":"fc23f483-c0dc-48d1-9a0c-5376be955b73","Type":"ContainerStarted","Data":"22ad7edaabd914d42156437e73507ccd1b24c6dc79d988a7f3f9b76889778d6f"} Sep 11 18:30:01.149078 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:01.148950 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-724327-29819190-dq289" event={"ID":"fc23f483-c0dc-48d1-9a0c-5376be955b73","Type":"ContainerStarted","Data":"661ada728967ec4e2d7364790c5738f3b454dd37570f4a1d0456806588137f23"} Sep 11 18:30:01.160790 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:01.160750 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-7e8ab9-29819190-7wmxb" podStartSLOduration=1.160735767 podStartE2EDuration="1.160735767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:30:01.157739079 +0000 UTC m=+882.654756833" watchObservedRunningTime="2026-09-11 18:30:01.160735767 +0000 UTC m=+882.657753521" Sep 11 18:30:01.173810 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:01.173763 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819190-q9l4g" podStartSLOduration=1.173746927 podStartE2EDuration="1.173746927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:30:01.171336802 +0000 UTC m=+882.668354560" watchObservedRunningTime="2026-09-11 18:30:01.173746927 +0000 UTC m=+882.670764680" Sep 11 18:30:01.187650 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:01.187599 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-724327-29819190-dq289" podStartSLOduration=1.187584427 podStartE2EDuration="1.187584427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:30:01.185710596 +0000 UTC m=+882.682728351" watchObservedRunningTime="2026-09-11 18:30:01.187584427 +0000 UTC m=+882.684602243" Sep 11 18:30:01.204442 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:01.204392 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-efa8b9-29819190-vjtwg" podStartSLOduration=1.204378275 podStartE2EDuration="1.204378275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:30:01.201861938 +0000 UTC m=+882.698879693" watchObservedRunningTime="2026-09-11 18:30:01.204378275 +0000 UTC m=+882.701396028" Sep 11 18:30:02.163874 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:02.163836 2561 generic.go:356] "Generic (PLEG): container finished" podID="fc23f483-c0dc-48d1-9a0c-5376be955b73" containerID="22ad7edaabd914d42156437e73507ccd1b24c6dc79d988a7f3f9b76889778d6f" exitCode=0 Sep 11 18:30:02.164308 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:02.163928 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-724327-29819190-dq289" event={"ID":"fc23f483-c0dc-48d1-9a0c-5376be955b73","Type":"ContainerDied","Data":"22ad7edaabd914d42156437e73507ccd1b24c6dc79d988a7f3f9b76889778d6f"} Sep 11 18:30:02.165665 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:02.165637 2561 generic.go:356] "Generic (PLEG): container finished" podID="66f53630-fd06-4546-aef0-b09d82db862b" containerID="a63bef399a98a4f6152d8ea5adae6399f6ddfd345155eec1acb8c4ef59bccf4e" exitCode=0 Sep 11 18:30:02.165790 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:02.165716 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-7e8ab9-29819190-7wmxb" event={"ID":"66f53630-fd06-4546-aef0-b09d82db862b","Type":"ContainerDied","Data":"a63bef399a98a4f6152d8ea5adae6399f6ddfd345155eec1acb8c4ef59bccf4e"} Sep 11 18:30:02.167438 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:02.167409 2561 generic.go:356] "Generic (PLEG): container finished" podID="079c554e-e599-4ba7-a259-0a5e8cafad30" containerID="6de739ac64002a6ef60650d5bfedd1f00cc9efef65a305dae1c3c14d14d6ec0d" exitCode=0 Sep 11 18:30:02.167539 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:02.167483 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-efa8b9-29819190-vjtwg" event={"ID":"079c554e-e599-4ba7-a259-0a5e8cafad30","Type":"ContainerDied","Data":"6de739ac64002a6ef60650d5bfedd1f00cc9efef65a305dae1c3c14d14d6ec0d"} Sep 11 18:30:02.168998 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:02.168979 2561 generic.go:356] "Generic (PLEG): container finished" podID="6c688bfb-48db-4bc1-839a-ee5f08c2994b" containerID="f47f5dfdc0b1e8fa96894fb4c2e78ef85508065ae0fe98b18a7a7f84a1aa3ba6" exitCode=0 Sep 11 18:30:02.169087 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:02.169001 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819190-q9l4g" event={"ID":"6c688bfb-48db-4bc1-839a-ee5f08c2994b","Type":"ContainerDied","Data":"f47f5dfdc0b1e8fa96894fb4c2e78ef85508065ae0fe98b18a7a7f84a1aa3ba6"} Sep 11 18:30:02.531554 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:02.531456 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68ac8212-xk4g7"] Sep 11 18:30:02.539590 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:02.539559 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68ac8212-xk4g7"] Sep 11 18:30:02.661233 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:02.661191 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc"] Sep 11 18:30:02.661556 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:02.661526 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc" podUID="77efe2e5-edb2-484f-98cb-9543b6d630d7" containerName="istio-proxy" containerID="cri-o://0ce0c34054fb290e7729520ac1232a1d32258986599e777540a232294a63edbd" gracePeriod=30 Sep 11 18:30:03.141372 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:03.141340 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52396ffe-7122-4a5a-8e4e-cf3ea4e87675" path="/var/lib/kubelet/pods/52396ffe-7122-4a5a-8e4e-cf3ea4e87675/volumes" Sep 11 18:30:03.343713 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:03.343679 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-7e8ab9-29819190-7wmxb" Sep 11 18:30:03.378772 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:03.378741 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819190-q9l4g" Sep 11 18:30:03.413956 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:03.413933 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-efa8b9-29819190-vjtwg" Sep 11 18:30:03.417448 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:03.417424 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-724327-29819190-dq289" Sep 11 18:30:04.188254 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:04.188223 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-efa8b9-29819190-vjtwg" Sep 11 18:30:04.188254 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:04.188245 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-efa8b9-29819190-vjtwg" event={"ID":"079c554e-e599-4ba7-a259-0a5e8cafad30","Type":"ContainerDied","Data":"e6454290dca5fbd15f43acc7fe0b9fa9a87401f91455f8f66a68f6ba1dfc9b9a"} Sep 11 18:30:04.188637 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:04.188297 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6454290dca5fbd15f43acc7fe0b9fa9a87401f91455f8f66a68f6ba1dfc9b9a" Sep 11 18:30:04.190020 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:04.189997 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819190-q9l4g" Sep 11 18:30:04.190192 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:04.190032 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819190-q9l4g" event={"ID":"6c688bfb-48db-4bc1-839a-ee5f08c2994b","Type":"ContainerDied","Data":"de6e104c6520502c86292c737b00c916e58dcfd5812d150d0a53df0827a0b7d3"} Sep 11 18:30:04.190192 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:04.190110 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de6e104c6520502c86292c737b00c916e58dcfd5812d150d0a53df0827a0b7d3" Sep 11 18:30:04.191736 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:04.191706 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-724327-29819190-dq289" Sep 11 18:30:04.191865 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:04.191734 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-724327-29819190-dq289" event={"ID":"fc23f483-c0dc-48d1-9a0c-5376be955b73","Type":"ContainerDied","Data":"661ada728967ec4e2d7364790c5738f3b454dd37570f4a1d0456806588137f23"} Sep 11 18:30:04.191865 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:04.191774 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="661ada728967ec4e2d7364790c5738f3b454dd37570f4a1d0456806588137f23" Sep 11 18:30:04.193306 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:04.193281 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-7e8ab9-29819190-7wmxb" event={"ID":"66f53630-fd06-4546-aef0-b09d82db862b","Type":"ContainerDied","Data":"1e3ee72610e50897dae036d68767f1585530fb1b033473e4fcd1087acf06ca95"} Sep 11 18:30:04.193450 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:04.193318 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-7e8ab9-29819190-7wmxb" Sep 11 18:30:04.193450 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:04.193321 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e3ee72610e50897dae036d68767f1585530fb1b033473e4fcd1087acf06ca95" Sep 11 18:30:07.920541 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:07.920515 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc" Sep 11 18:30:07.952801 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:07.952730 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77efe2e5-edb2-484f-98cb-9543b6d630d7-istio-data\") pod \"77efe2e5-edb2-484f-98cb-9543b6d630d7\" (UID: \"77efe2e5-edb2-484f-98cb-9543b6d630d7\") " Sep 11 18:30:07.952801 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:07.952777 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77efe2e5-edb2-484f-98cb-9543b6d630d7-istiod-ca-cert\") pod \"77efe2e5-edb2-484f-98cb-9543b6d630d7\" (UID: \"77efe2e5-edb2-484f-98cb-9543b6d630d7\") " Sep 11 18:30:07.952801 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:07.952795 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77efe2e5-edb2-484f-98cb-9543b6d630d7-istio-envoy\") pod \"77efe2e5-edb2-484f-98cb-9543b6d630d7\" (UID: \"77efe2e5-edb2-484f-98cb-9543b6d630d7\") " Sep 11 18:30:07.953019 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:07.952862 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rkrpl\" (UniqueName: \"kubernetes.io/projected/77efe2e5-edb2-484f-98cb-9543b6d630d7-kube-api-access-rkrpl\") pod \"77efe2e5-edb2-484f-98cb-9543b6d630d7\" (UID: \"77efe2e5-edb2-484f-98cb-9543b6d630d7\") " Sep 11 18:30:07.953019 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:07.952900 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77efe2e5-edb2-484f-98cb-9543b6d630d7-istio-podinfo\") pod \"77efe2e5-edb2-484f-98cb-9543b6d630d7\" (UID: \"77efe2e5-edb2-484f-98cb-9543b6d630d7\") " Sep 11 18:30:07.953019 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:07.952940 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77efe2e5-edb2-484f-98cb-9543b6d630d7-istio-token\") pod \"77efe2e5-edb2-484f-98cb-9543b6d630d7\" (UID: \"77efe2e5-edb2-484f-98cb-9543b6d630d7\") " Sep 11 18:30:07.953019 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:07.952967 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77efe2e5-edb2-484f-98cb-9543b6d630d7-workload-certs\") pod \"77efe2e5-edb2-484f-98cb-9543b6d630d7\" (UID: \"77efe2e5-edb2-484f-98cb-9543b6d630d7\") " Sep 11 18:30:07.953019 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:07.953004 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77efe2e5-edb2-484f-98cb-9543b6d630d7-workload-socket\") pod \"77efe2e5-edb2-484f-98cb-9543b6d630d7\" (UID: \"77efe2e5-edb2-484f-98cb-9543b6d630d7\") " Sep 11 18:30:07.953317 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:07.953032 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77efe2e5-edb2-484f-98cb-9543b6d630d7-istio-data" (OuterVolumeSpecName: "istio-data") pod "77efe2e5-edb2-484f-98cb-9543b6d630d7" (UID: "77efe2e5-edb2-484f-98cb-9543b6d630d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:30:07.953317 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:07.953114 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77efe2e5-edb2-484f-98cb-9543b6d630d7-credential-socket\") pod \"77efe2e5-edb2-484f-98cb-9543b6d630d7\" (UID: \"77efe2e5-edb2-484f-98cb-9543b6d630d7\") " Sep 11 18:30:07.953317 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:07.953187 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77efe2e5-edb2-484f-98cb-9543b6d630d7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "77efe2e5-edb2-484f-98cb-9543b6d630d7" (UID: "77efe2e5-edb2-484f-98cb-9543b6d630d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:30:07.953478 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:07.953381 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77efe2e5-edb2-484f-98cb-9543b6d630d7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "77efe2e5-edb2-484f-98cb-9543b6d630d7" (UID: "77efe2e5-edb2-484f-98cb-9543b6d630d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:30:07.953534 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:07.953470 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77efe2e5-edb2-484f-98cb-9543b6d630d7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "77efe2e5-edb2-484f-98cb-9543b6d630d7" (UID: "77efe2e5-edb2-484f-98cb-9543b6d630d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:30:07.953587 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:07.953532 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77efe2e5-edb2-484f-98cb-9543b6d630d7-istio-data\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:30:07.953587 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:07.953552 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77efe2e5-edb2-484f-98cb-9543b6d630d7-istiod-ca-cert\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:30:07.953587 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:07.953566 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77efe2e5-edb2-484f-98cb-9543b6d630d7-workload-certs\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:30:07.953736 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:07.953639 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77efe2e5-edb2-484f-98cb-9543b6d630d7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "77efe2e5-edb2-484f-98cb-9543b6d630d7" (UID: "77efe2e5-edb2-484f-98cb-9543b6d630d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:30:07.955256 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:07.955228 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77efe2e5-edb2-484f-98cb-9543b6d630d7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "77efe2e5-edb2-484f-98cb-9543b6d630d7" (UID: "77efe2e5-edb2-484f-98cb-9543b6d630d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 18:30:07.955361 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:07.955254 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77efe2e5-edb2-484f-98cb-9543b6d630d7-kube-api-access-rkrpl" (OuterVolumeSpecName: "kube-api-access-rkrpl") pod "77efe2e5-edb2-484f-98cb-9543b6d630d7" (UID: "77efe2e5-edb2-484f-98cb-9543b6d630d7"). InnerVolumeSpecName "kube-api-access-rkrpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:30:07.955521 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:07.955505 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77efe2e5-edb2-484f-98cb-9543b6d630d7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "77efe2e5-edb2-484f-98cb-9543b6d630d7" (UID: "77efe2e5-edb2-484f-98cb-9543b6d630d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:30:07.955563 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:07.955519 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77efe2e5-edb2-484f-98cb-9543b6d630d7-istio-token" (OuterVolumeSpecName: "istio-token") pod "77efe2e5-edb2-484f-98cb-9543b6d630d7" (UID: "77efe2e5-edb2-484f-98cb-9543b6d630d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:30:08.054080 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:08.054032 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77efe2e5-edb2-484f-98cb-9543b6d630d7-credential-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:30:08.054080 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:08.054083 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77efe2e5-edb2-484f-98cb-9543b6d630d7-istio-envoy\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:30:08.054277 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:08.054093 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkrpl\" (UniqueName: \"kubernetes.io/projected/77efe2e5-edb2-484f-98cb-9543b6d630d7-kube-api-access-rkrpl\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:30:08.054277 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:08.054101 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77efe2e5-edb2-484f-98cb-9543b6d630d7-istio-podinfo\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:30:08.054277 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:08.054111 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77efe2e5-edb2-484f-98cb-9543b6d630d7-istio-token\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:30:08.054277 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:08.054119 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77efe2e5-edb2-484f-98cb-9543b6d630d7-workload-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:30:08.223743 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:08.223648 2561 generic.go:356] "Generic (PLEG): container finished" podID="77efe2e5-edb2-484f-98cb-9543b6d630d7" containerID="0ce0c34054fb290e7729520ac1232a1d32258986599e777540a232294a63edbd" exitCode=0 Sep 11 18:30:08.223743 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:08.223702 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc" Sep 11 18:30:08.223743 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:08.223724 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc" event={"ID":"77efe2e5-edb2-484f-98cb-9543b6d630d7","Type":"ContainerDied","Data":"0ce0c34054fb290e7729520ac1232a1d32258986599e777540a232294a63edbd"} Sep 11 18:30:08.223994 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:08.223792 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc" event={"ID":"77efe2e5-edb2-484f-98cb-9543b6d630d7","Type":"ContainerDied","Data":"cd23267e739998b58e9a333e39716f4f85241990719d5207f97ddad5bb0a3d6a"} Sep 11 18:30:08.223994 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:08.223818 2561 scope.go:117] "RemoveContainer" containerID="0ce0c34054fb290e7729520ac1232a1d32258986599e777540a232294a63edbd" Sep 11 18:30:08.234031 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:08.234014 2561 scope.go:117] "RemoveContainer" containerID="f272535e50c5914ca2c847b649a4bc914f0d1dfd07ecbeb5eb755712d78b62c3" Sep 11 18:30:08.243020 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:08.243001 2561 scope.go:117] "RemoveContainer" containerID="0ce0c34054fb290e7729520ac1232a1d32258986599e777540a232294a63edbd" Sep 11 18:30:08.243284 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:30:08.243267 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ce0c34054fb290e7729520ac1232a1d32258986599e777540a232294a63edbd\": container with ID starting with 0ce0c34054fb290e7729520ac1232a1d32258986599e777540a232294a63edbd not found: ID does not exist" containerID="0ce0c34054fb290e7729520ac1232a1d32258986599e777540a232294a63edbd" Sep 11 18:30:08.243336 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:08.243291 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ce0c34054fb290e7729520ac1232a1d32258986599e777540a232294a63edbd"} err="failed to get container status \"0ce0c34054fb290e7729520ac1232a1d32258986599e777540a232294a63edbd\": rpc error: code = NotFound desc = could not find container \"0ce0c34054fb290e7729520ac1232a1d32258986599e777540a232294a63edbd\": container with ID starting with 0ce0c34054fb290e7729520ac1232a1d32258986599e777540a232294a63edbd not found: ID does not exist" Sep 11 18:30:08.243336 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:08.243306 2561 scope.go:117] "RemoveContainer" containerID="f272535e50c5914ca2c847b649a4bc914f0d1dfd07ecbeb5eb755712d78b62c3" Sep 11 18:30:08.243525 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:30:08.243504 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f272535e50c5914ca2c847b649a4bc914f0d1dfd07ecbeb5eb755712d78b62c3\": container with ID starting with f272535e50c5914ca2c847b649a4bc914f0d1dfd07ecbeb5eb755712d78b62c3 not found: ID does not exist" containerID="f272535e50c5914ca2c847b649a4bc914f0d1dfd07ecbeb5eb755712d78b62c3" Sep 11 18:30:08.243567 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:08.243531 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f272535e50c5914ca2c847b649a4bc914f0d1dfd07ecbeb5eb755712d78b62c3"} err="failed to get container status \"f272535e50c5914ca2c847b649a4bc914f0d1dfd07ecbeb5eb755712d78b62c3\": rpc error: code = NotFound desc = could not find container \"f272535e50c5914ca2c847b649a4bc914f0d1dfd07ecbeb5eb755712d78b62c3\": container with ID starting with f272535e50c5914ca2c847b649a4bc914f0d1dfd07ecbeb5eb755712d78b62c3 not found: ID does not exist" Sep 11 18:30:08.248308 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:08.248284 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc"] Sep 11 18:30:08.252406 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:08.252382 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-68ac8212-openshift-default-685984cd7b-4pfmc"] Sep 11 18:30:09.141745 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:09.141661 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77efe2e5-edb2-484f-98cb-9543b6d630d7" path="/var/lib/kubelet/pods/77efe2e5-edb2-484f-98cb-9543b6d630d7/volumes" Sep 11 18:30:12.511139 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:12.511062 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7e8ab9-vhqb2"] Sep 11 18:30:12.511550 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:12.511534 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77efe2e5-edb2-484f-98cb-9543b6d630d7" containerName="istio-proxy" Sep 11 18:30:12.511594 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:12.511552 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="77efe2e5-edb2-484f-98cb-9543b6d630d7" containerName="istio-proxy" Sep 11 18:30:12.511594 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:12.511561 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc23f483-c0dc-48d1-9a0c-5376be955b73" containerName="cleanup" Sep 11 18:30:12.511594 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:12.511568 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc23f483-c0dc-48d1-9a0c-5376be955b73" containerName="cleanup" Sep 11 18:30:12.511594 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:12.511581 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66f53630-fd06-4546-aef0-b09d82db862b" containerName="cleanup" Sep 11 18:30:12.511594 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:12.511587 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="66f53630-fd06-4546-aef0-b09d82db862b" containerName="cleanup" Sep 11 18:30:12.511750 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:12.511597 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c688bfb-48db-4bc1-839a-ee5f08c2994b" containerName="cleanup" Sep 11 18:30:12.511750 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:12.511603 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c688bfb-48db-4bc1-839a-ee5f08c2994b" containerName="cleanup" Sep 11 18:30:12.511750 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:12.511609 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="079c554e-e599-4ba7-a259-0a5e8cafad30" containerName="cleanup" Sep 11 18:30:12.511750 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:12.511614 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="079c554e-e599-4ba7-a259-0a5e8cafad30" containerName="cleanup" Sep 11 18:30:12.511750 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:12.511692 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="66f53630-fd06-4546-aef0-b09d82db862b" containerName="cleanup" Sep 11 18:30:12.511750 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:12.511703 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="079c554e-e599-4ba7-a259-0a5e8cafad30" containerName="cleanup" Sep 11 18:30:12.511750 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:12.511710 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="77efe2e5-edb2-484f-98cb-9543b6d630d7" containerName="istio-proxy" Sep 11 18:30:12.511750 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:12.511717 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc23f483-c0dc-48d1-9a0c-5376be955b73" containerName="cleanup" Sep 11 18:30:12.511750 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:12.511725 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c688bfb-48db-4bc1-839a-ee5f08c2994b" containerName="cleanup" Sep 11 18:30:12.516393 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:12.516366 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7e8ab9-vhqb2" Sep 11 18:30:12.519160 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:12.519119 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 18:30:12.519352 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:12.519129 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n968b\"" Sep 11 18:30:12.522099 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:12.522067 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7e8ab9-vhqb2"] Sep 11 18:30:12.602617 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:12.602583 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db560cf0-a1b1-41cd-98fa-01a5a33d4831-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-7e8ab9-vhqb2\" (UID: \"db560cf0-a1b1-41cd-98fa-01a5a33d4831\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7e8ab9-vhqb2" Sep 11 18:30:12.703880 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:12.703843 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db560cf0-a1b1-41cd-98fa-01a5a33d4831-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-7e8ab9-vhqb2\" (UID: \"db560cf0-a1b1-41cd-98fa-01a5a33d4831\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7e8ab9-vhqb2" Sep 11 18:30:12.704508 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:12.704478 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db560cf0-a1b1-41cd-98fa-01a5a33d4831-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-7e8ab9-vhqb2\" (UID: \"db560cf0-a1b1-41cd-98fa-01a5a33d4831\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7e8ab9-vhqb2" Sep 11 18:30:12.830123 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:12.830004 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7e8ab9-vhqb2" Sep 11 18:30:13.111516 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:13.111474 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7e8ab9-vhqb2"] Sep 11 18:30:13.114036 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:30:13.113993 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb560cf0_a1b1_41cd_98fa_01a5a33d4831.slice/crio-70f9e5d30cdeef34d6d135147f4cf852f457946ef010cd928f6c28566e926f37 WatchSource:0}: Error finding container 70f9e5d30cdeef34d6d135147f4cf852f457946ef010cd928f6c28566e926f37: Status 404 returned error can't find the container with id 70f9e5d30cdeef34d6d135147f4cf852f457946ef010cd928f6c28566e926f37 Sep 11 18:30:13.257977 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:13.257936 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7e8ab9-vhqb2" event={"ID":"db560cf0-a1b1-41cd-98fa-01a5a33d4831","Type":"ContainerStarted","Data":"70f9e5d30cdeef34d6d135147f4cf852f457946ef010cd928f6c28566e926f37"} Sep 11 18:30:14.268635 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:14.268603 2561 generic.go:356] "Generic (PLEG): container finished" podID="db560cf0-a1b1-41cd-98fa-01a5a33d4831" containerID="04e92e3d37eab8ebcd55d40fdbfc31b381cc5bf8082520aacd829499500503b1" exitCode=0 Sep 11 18:30:14.269038 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:14.268692 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7e8ab9-vhqb2" event={"ID":"db560cf0-a1b1-41cd-98fa-01a5a33d4831","Type":"ContainerDied","Data":"04e92e3d37eab8ebcd55d40fdbfc31b381cc5bf8082520aacd829499500503b1"} Sep 11 18:30:15.425998 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:15.425967 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7e8ab9-vhqb2" Sep 11 18:30:15.533280 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:15.533178 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db560cf0-a1b1-41cd-98fa-01a5a33d4831-maas-api-service-ca\") pod \"db560cf0-a1b1-41cd-98fa-01a5a33d4831\" (UID: \"db560cf0-a1b1-41cd-98fa-01a5a33d4831\") " Sep 11 18:30:15.533549 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:15.533519 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db560cf0-a1b1-41cd-98fa-01a5a33d4831-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "db560cf0-a1b1-41cd-98fa-01a5a33d4831" (UID: "db560cf0-a1b1-41cd-98fa-01a5a33d4831"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:30:15.634470 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:15.634441 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db560cf0-a1b1-41cd-98fa-01a5a33d4831-maas-api-service-ca\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:30:16.284855 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:16.284821 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7e8ab9-vhqb2" Sep 11 18:30:16.285132 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:16.284858 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7e8ab9-vhqb2" event={"ID":"db560cf0-a1b1-41cd-98fa-01a5a33d4831","Type":"ContainerDied","Data":"70f9e5d30cdeef34d6d135147f4cf852f457946ef010cd928f6c28566e926f37"} Sep 11 18:30:16.285132 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:16.284918 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70f9e5d30cdeef34d6d135147f4cf852f457946ef010cd928f6c28566e926f37" Sep 11 18:30:19.368517 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:19.368479 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dj524_60702ef3-98ef-463b-b877-3a00825353d8/console-operator/1.log" Sep 11 18:30:19.371469 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:19.371445 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dj524_60702ef3-98ef-463b-b877-3a00825353d8/console-operator/1.log" Sep 11 18:30:19.377121 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:19.377095 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4knd_42e5ddd8-a96c-45a1-adf9-caa47ee902db/ovn-acl-logging/0.log" Sep 11 18:30:19.380221 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:19.380202 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4knd_42e5ddd8-a96c-45a1-adf9-caa47ee902db/ovn-acl-logging/0.log" Sep 11 18:30:23.100358 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:23.100324 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-7e8ab9-29819190-7wmxb"] Sep 11 18:30:23.103086 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:23.103048 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-7e8ab9-29819190-7wmxb"] Sep 11 18:30:23.141170 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:23.141124 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66f53630-fd06-4546-aef0-b09d82db862b" path="/var/lib/kubelet/pods/66f53630-fd06-4546-aef0-b09d82db862b/volumes" Sep 11 18:30:23.369838 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:23.369742 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7e8ab9-vhqb2"] Sep 11 18:30:23.373788 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:23.373757 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7e8ab9-vhqb2"] Sep 11 18:30:25.139589 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:30:25.139552 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db560cf0-a1b1-41cd-98fa-01a5a33d4831" path="/var/lib/kubelet/pods/db560cf0-a1b1-41cd-98fa-01a5a33d4831/volumes" Sep 11 18:31:06.804668 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:06.804626 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn"] Sep 11 18:31:06.805270 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:06.804929 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn" podUID="3730018e-ef6b-4210-9007-0861b85202df" containerName="istio-proxy" containerID="cri-o://fd43669314cad07c05d63f7aa69d80d018c221a827b7edda4f013bfc919c1606" gracePeriod=30 Sep 11 18:31:07.546194 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:07.546155 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm"] Sep 11 18:31:07.546843 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:07.546817 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db560cf0-a1b1-41cd-98fa-01a5a33d4831" containerName="revoke-keys" Sep 11 18:31:07.546985 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:07.546860 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="db560cf0-a1b1-41cd-98fa-01a5a33d4831" containerName="revoke-keys" Sep 11 18:31:07.546985 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:07.546884 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77efe2e5-edb2-484f-98cb-9543b6d630d7" containerName="istio-proxy" Sep 11 18:31:07.546985 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:07.546893 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="77efe2e5-edb2-484f-98cb-9543b6d630d7" containerName="istio-proxy" Sep 11 18:31:07.547176 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:07.547046 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="77efe2e5-edb2-484f-98cb-9543b6d630d7" containerName="istio-proxy" Sep 11 18:31:07.547176 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:07.547086 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="db560cf0-a1b1-41cd-98fa-01a5a33d4831" containerName="revoke-keys" Sep 11 18:31:07.550410 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:07.550388 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm" Sep 11 18:31:07.552647 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:07.552624 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-c7c83f-openshift-default-dockercfg-78np7\"" Sep 11 18:31:07.565586 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:07.565556 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm"] Sep 11 18:31:07.672606 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:07.672569 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ab46e7b-3161-46a7-9603-629d75d31c1d-workload-certs\") pod \"e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm\" (UID: \"1ab46e7b-3161-46a7-9603-629d75d31c1d\") " pod="openshift-ingress/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm" Sep 11 18:31:07.672782 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:07.672614 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ab46e7b-3161-46a7-9603-629d75d31c1d-istio-podinfo\") pod \"e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm\" (UID: \"1ab46e7b-3161-46a7-9603-629d75d31c1d\") " pod="openshift-ingress/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm" Sep 11 18:31:07.672782 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:07.672634 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdgt8\" (UniqueName: \"kubernetes.io/projected/1ab46e7b-3161-46a7-9603-629d75d31c1d-kube-api-access-vdgt8\") pod \"e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm\" (UID: \"1ab46e7b-3161-46a7-9603-629d75d31c1d\") " pod="openshift-ingress/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm" Sep 11 18:31:07.672782 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:07.672659 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ab46e7b-3161-46a7-9603-629d75d31c1d-workload-socket\") pod \"e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm\" (UID: \"1ab46e7b-3161-46a7-9603-629d75d31c1d\") " pod="openshift-ingress/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm" Sep 11 18:31:07.672782 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:07.672686 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ab46e7b-3161-46a7-9603-629d75d31c1d-istio-envoy\") pod \"e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm\" (UID: \"1ab46e7b-3161-46a7-9603-629d75d31c1d\") " pod="openshift-ingress/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm" Sep 11 18:31:07.672782 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:07.672745 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ab46e7b-3161-46a7-9603-629d75d31c1d-istio-token\") pod \"e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm\" (UID: \"1ab46e7b-3161-46a7-9603-629d75d31c1d\") " pod="openshift-ingress/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm" Sep 11 18:31:07.672951 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:07.672804 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ab46e7b-3161-46a7-9603-629d75d31c1d-credential-socket\") pod \"e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm\" (UID: \"1ab46e7b-3161-46a7-9603-629d75d31c1d\") " pod="openshift-ingress/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm" Sep 11 18:31:07.672951 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:07.672837 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ab46e7b-3161-46a7-9603-629d75d31c1d-istio-data\") pod \"e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm\" (UID: \"1ab46e7b-3161-46a7-9603-629d75d31c1d\") " pod="openshift-ingress/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm" Sep 11 18:31:07.672951 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:07.672891 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ab46e7b-3161-46a7-9603-629d75d31c1d-istiod-ca-cert\") pod \"e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm\" (UID: \"1ab46e7b-3161-46a7-9603-629d75d31c1d\") " pod="openshift-ingress/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm" Sep 11 18:31:07.774212 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:07.774171 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ab46e7b-3161-46a7-9603-629d75d31c1d-workload-certs\") pod \"e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm\" (UID: \"1ab46e7b-3161-46a7-9603-629d75d31c1d\") " pod="openshift-ingress/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm" Sep 11 18:31:07.774212 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:07.774220 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ab46e7b-3161-46a7-9603-629d75d31c1d-istio-podinfo\") pod \"e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm\" (UID: \"1ab46e7b-3161-46a7-9603-629d75d31c1d\") " pod="openshift-ingress/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm" Sep 11 18:31:07.774495 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:07.774238 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdgt8\" (UniqueName: \"kubernetes.io/projected/1ab46e7b-3161-46a7-9603-629d75d31c1d-kube-api-access-vdgt8\") pod \"e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm\" (UID: \"1ab46e7b-3161-46a7-9603-629d75d31c1d\") " pod="openshift-ingress/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm" Sep 11 18:31:07.774495 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:07.774478 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ab46e7b-3161-46a7-9603-629d75d31c1d-workload-socket\") pod \"e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm\" (UID: \"1ab46e7b-3161-46a7-9603-629d75d31c1d\") " pod="openshift-ingress/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm" Sep 11 18:31:07.774612 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:07.774533 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ab46e7b-3161-46a7-9603-629d75d31c1d-istio-envoy\") pod \"e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm\" (UID: \"1ab46e7b-3161-46a7-9603-629d75d31c1d\") " pod="openshift-ingress/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm" Sep 11 18:31:07.774612 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:07.774569 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ab46e7b-3161-46a7-9603-629d75d31c1d-istio-token\") pod \"e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm\" (UID: \"1ab46e7b-3161-46a7-9603-629d75d31c1d\") " pod="openshift-ingress/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm" Sep 11 18:31:07.774612 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:07.774597 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ab46e7b-3161-46a7-9603-629d75d31c1d-credential-socket\") pod \"e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm\" (UID: \"1ab46e7b-3161-46a7-9603-629d75d31c1d\") " pod="openshift-ingress/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm" Sep 11 18:31:07.774791 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:07.774654 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ab46e7b-3161-46a7-9603-629d75d31c1d-istio-data\") pod \"e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm\" (UID: \"1ab46e7b-3161-46a7-9603-629d75d31c1d\") " pod="openshift-ingress/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm" Sep 11 18:31:07.774791 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:07.774666 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ab46e7b-3161-46a7-9603-629d75d31c1d-workload-certs\") pod \"e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm\" (UID: \"1ab46e7b-3161-46a7-9603-629d75d31c1d\") " pod="openshift-ingress/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm" Sep 11 18:31:07.774791 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:07.774726 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ab46e7b-3161-46a7-9603-629d75d31c1d-istiod-ca-cert\") pod \"e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm\" (UID: \"1ab46e7b-3161-46a7-9603-629d75d31c1d\") " pod="openshift-ingress/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm" Sep 11 18:31:07.774938 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:07.774848 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ab46e7b-3161-46a7-9603-629d75d31c1d-workload-socket\") pod \"e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm\" (UID: \"1ab46e7b-3161-46a7-9603-629d75d31c1d\") " pod="openshift-ingress/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm" Sep 11 18:31:07.774998 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:07.774964 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ab46e7b-3161-46a7-9603-629d75d31c1d-istio-data\") pod \"e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm\" (UID: \"1ab46e7b-3161-46a7-9603-629d75d31c1d\") " pod="openshift-ingress/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm" Sep 11 18:31:07.775093 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:07.775008 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ab46e7b-3161-46a7-9603-629d75d31c1d-credential-socket\") pod \"e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm\" (UID: \"1ab46e7b-3161-46a7-9603-629d75d31c1d\") " pod="openshift-ingress/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm" Sep 11 18:31:07.775405 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:07.775384 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ab46e7b-3161-46a7-9603-629d75d31c1d-istiod-ca-cert\") pod \"e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm\" (UID: \"1ab46e7b-3161-46a7-9603-629d75d31c1d\") " pod="openshift-ingress/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm" Sep 11 18:31:07.776744 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:07.776719 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ab46e7b-3161-46a7-9603-629d75d31c1d-istio-envoy\") pod \"e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm\" (UID: \"1ab46e7b-3161-46a7-9603-629d75d31c1d\") " pod="openshift-ingress/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm" Sep 11 18:31:07.776860 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:07.776843 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ab46e7b-3161-46a7-9603-629d75d31c1d-istio-podinfo\") pod \"e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm\" (UID: \"1ab46e7b-3161-46a7-9603-629d75d31c1d\") " pod="openshift-ingress/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm" Sep 11 18:31:07.783911 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:07.783875 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ab46e7b-3161-46a7-9603-629d75d31c1d-istio-token\") pod \"e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm\" (UID: \"1ab46e7b-3161-46a7-9603-629d75d31c1d\") " pod="openshift-ingress/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm" Sep 11 18:31:07.784027 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:07.784003 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdgt8\" (UniqueName: \"kubernetes.io/projected/1ab46e7b-3161-46a7-9603-629d75d31c1d-kube-api-access-vdgt8\") pod \"e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm\" (UID: \"1ab46e7b-3161-46a7-9603-629d75d31c1d\") " pod="openshift-ingress/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm" Sep 11 18:31:07.862432 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:07.862350 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm" Sep 11 18:31:08.153568 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:08.153535 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm"] Sep 11 18:31:08.154649 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:31:08.154620 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ab46e7b_3161_46a7_9603_629d75d31c1d.slice/crio-11a82d7813c712b6b9fb8876147ef51790d11729565d53111fb7709f46926296 WatchSource:0}: Error finding container 11a82d7813c712b6b9fb8876147ef51790d11729565d53111fb7709f46926296: Status 404 returned error can't find the container with id 11a82d7813c712b6b9fb8876147ef51790d11729565d53111fb7709f46926296 Sep 11 18:31:08.156783 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:08.156744 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:31:08.156897 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:08.156813 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:31:08.156897 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:08.156839 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:31:08.636987 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:08.636944 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm" event={"ID":"1ab46e7b-3161-46a7-9603-629d75d31c1d","Type":"ContainerStarted","Data":"d108db93c1153fe7bcaa3cacfa3d4c4298feeacdf22a149ef107297d56e356e6"} Sep 11 18:31:08.637206 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:08.637008 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm" event={"ID":"1ab46e7b-3161-46a7-9603-629d75d31c1d","Type":"ContainerStarted","Data":"11a82d7813c712b6b9fb8876147ef51790d11729565d53111fb7709f46926296"} Sep 11 18:31:08.663964 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:08.663910 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm" podStartSLOduration=1.6638934619999999 podStartE2EDuration="1.663893462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:31:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:31:08.663420706 +0000 UTC m=+950.160438461" watchObservedRunningTime="2026-09-11 18:31:08.663893462 +0000 UTC m=+950.160911216" Sep 11 18:31:08.863240 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:08.863198 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm" Sep 11 18:31:09.868949 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:09.868914 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm" Sep 11 18:31:10.650208 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:10.650179 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm" Sep 11 18:31:10.651207 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:10.651181 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm" Sep 11 18:31:12.051536 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:12.051507 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn" Sep 11 18:31:12.223011 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:12.222977 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3730018e-ef6b-4210-9007-0861b85202df-workload-socket\") pod \"3730018e-ef6b-4210-9007-0861b85202df\" (UID: \"3730018e-ef6b-4210-9007-0861b85202df\") " Sep 11 18:31:12.223239 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:12.223033 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3730018e-ef6b-4210-9007-0861b85202df-istio-envoy\") pod \"3730018e-ef6b-4210-9007-0861b85202df\" (UID: \"3730018e-ef6b-4210-9007-0861b85202df\") " Sep 11 18:31:12.223239 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:12.223086 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-78f9q\" (UniqueName: \"kubernetes.io/projected/3730018e-ef6b-4210-9007-0861b85202df-kube-api-access-78f9q\") pod \"3730018e-ef6b-4210-9007-0861b85202df\" (UID: \"3730018e-ef6b-4210-9007-0861b85202df\") " Sep 11 18:31:12.223239 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:12.223150 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3730018e-ef6b-4210-9007-0861b85202df-workload-certs\") pod \"3730018e-ef6b-4210-9007-0861b85202df\" (UID: \"3730018e-ef6b-4210-9007-0861b85202df\") " Sep 11 18:31:12.223239 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:12.223190 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3730018e-ef6b-4210-9007-0861b85202df-credential-socket\") pod \"3730018e-ef6b-4210-9007-0861b85202df\" (UID: \"3730018e-ef6b-4210-9007-0861b85202df\") " Sep 11 18:31:12.223239 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:12.223225 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3730018e-ef6b-4210-9007-0861b85202df-istio-token\") pod \"3730018e-ef6b-4210-9007-0861b85202df\" (UID: \"3730018e-ef6b-4210-9007-0861b85202df\") " Sep 11 18:31:12.223519 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:12.223287 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3730018e-ef6b-4210-9007-0861b85202df-istiod-ca-cert\") pod \"3730018e-ef6b-4210-9007-0861b85202df\" (UID: \"3730018e-ef6b-4210-9007-0861b85202df\") " Sep 11 18:31:12.223519 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:12.223325 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3730018e-ef6b-4210-9007-0861b85202df-istio-data\") pod \"3730018e-ef6b-4210-9007-0861b85202df\" (UID: \"3730018e-ef6b-4210-9007-0861b85202df\") " Sep 11 18:31:12.223519 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:12.223338 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3730018e-ef6b-4210-9007-0861b85202df-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3730018e-ef6b-4210-9007-0861b85202df" (UID: "3730018e-ef6b-4210-9007-0861b85202df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:31:12.223519 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:12.223356 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3730018e-ef6b-4210-9007-0861b85202df-istio-podinfo\") pod \"3730018e-ef6b-4210-9007-0861b85202df\" (UID: \"3730018e-ef6b-4210-9007-0861b85202df\") " Sep 11 18:31:12.223519 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:12.223477 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3730018e-ef6b-4210-9007-0861b85202df-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3730018e-ef6b-4210-9007-0861b85202df" (UID: "3730018e-ef6b-4210-9007-0861b85202df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:31:12.223772 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:12.223649 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3730018e-ef6b-4210-9007-0861b85202df-istio-data" (OuterVolumeSpecName: "istio-data") pod "3730018e-ef6b-4210-9007-0861b85202df" (UID: "3730018e-ef6b-4210-9007-0861b85202df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:31:12.223772 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:12.223694 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3730018e-ef6b-4210-9007-0861b85202df-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3730018e-ef6b-4210-9007-0861b85202df" (UID: "3730018e-ef6b-4210-9007-0861b85202df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:31:12.223904 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:12.223883 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3730018e-ef6b-4210-9007-0861b85202df-workload-certs\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:31:12.223969 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:12.223910 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3730018e-ef6b-4210-9007-0861b85202df-istiod-ca-cert\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:31:12.224028 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:12.223993 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3730018e-ef6b-4210-9007-0861b85202df-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3730018e-ef6b-4210-9007-0861b85202df" (UID: "3730018e-ef6b-4210-9007-0861b85202df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:31:12.224109 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:12.224038 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3730018e-ef6b-4210-9007-0861b85202df-istio-data\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:31:12.224109 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:12.224070 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3730018e-ef6b-4210-9007-0861b85202df-workload-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:31:12.225413 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:12.225388 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3730018e-ef6b-4210-9007-0861b85202df-kube-api-access-78f9q" (OuterVolumeSpecName: "kube-api-access-78f9q") pod "3730018e-ef6b-4210-9007-0861b85202df" (UID: "3730018e-ef6b-4210-9007-0861b85202df"). InnerVolumeSpecName "kube-api-access-78f9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:31:12.225555 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:12.225538 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3730018e-ef6b-4210-9007-0861b85202df-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3730018e-ef6b-4210-9007-0861b85202df" (UID: "3730018e-ef6b-4210-9007-0861b85202df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 18:31:12.225678 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:12.225662 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3730018e-ef6b-4210-9007-0861b85202df-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3730018e-ef6b-4210-9007-0861b85202df" (UID: "3730018e-ef6b-4210-9007-0861b85202df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:31:12.225736 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:12.225693 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3730018e-ef6b-4210-9007-0861b85202df-istio-token" (OuterVolumeSpecName: "istio-token") pod "3730018e-ef6b-4210-9007-0861b85202df" (UID: "3730018e-ef6b-4210-9007-0861b85202df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:31:12.325081 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:12.325016 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3730018e-ef6b-4210-9007-0861b85202df-istio-envoy\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:31:12.325081 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:12.325086 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78f9q\" (UniqueName: \"kubernetes.io/projected/3730018e-ef6b-4210-9007-0861b85202df-kube-api-access-78f9q\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:31:12.325294 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:12.325104 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3730018e-ef6b-4210-9007-0861b85202df-credential-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:31:12.325294 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:12.325117 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3730018e-ef6b-4210-9007-0861b85202df-istio-token\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:31:12.325294 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:12.325134 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3730018e-ef6b-4210-9007-0861b85202df-istio-podinfo\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:31:12.665200 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:12.665162 2561 generic.go:356] "Generic (PLEG): container finished" podID="3730018e-ef6b-4210-9007-0861b85202df" containerID="fd43669314cad07c05d63f7aa69d80d018c221a827b7edda4f013bfc919c1606" exitCode=0 Sep 11 18:31:12.665367 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:12.665213 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn" Sep 11 18:31:12.665367 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:12.665249 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn" event={"ID":"3730018e-ef6b-4210-9007-0861b85202df","Type":"ContainerDied","Data":"fd43669314cad07c05d63f7aa69d80d018c221a827b7edda4f013bfc919c1606"} Sep 11 18:31:12.665367 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:12.665303 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn" event={"ID":"3730018e-ef6b-4210-9007-0861b85202df","Type":"ContainerDied","Data":"d2084a2870191eaf91fa9c528b9b3ae2a4d345c23f0141ccc38d3f34e3a7c2dd"} Sep 11 18:31:12.665367 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:12.665328 2561 scope.go:117] "RemoveContainer" containerID="fd43669314cad07c05d63f7aa69d80d018c221a827b7edda4f013bfc919c1606" Sep 11 18:31:12.683590 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:12.683560 2561 scope.go:117] "RemoveContainer" containerID="966aca739243da692077f59bc34833a01cdffbc4013aa25842441a919628b33d" Sep 11 18:31:12.694349 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:12.694319 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn"] Sep 11 18:31:12.695323 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:12.695090 2561 scope.go:117] "RemoveContainer" containerID="fd43669314cad07c05d63f7aa69d80d018c221a827b7edda4f013bfc919c1606" Sep 11 18:31:12.695604 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:31:12.695582 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd43669314cad07c05d63f7aa69d80d018c221a827b7edda4f013bfc919c1606\": container with ID starting with fd43669314cad07c05d63f7aa69d80d018c221a827b7edda4f013bfc919c1606 not found: ID does not exist" containerID="fd43669314cad07c05d63f7aa69d80d018c221a827b7edda4f013bfc919c1606" Sep 11 18:31:12.695658 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:12.695614 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd43669314cad07c05d63f7aa69d80d018c221a827b7edda4f013bfc919c1606"} err="failed to get container status \"fd43669314cad07c05d63f7aa69d80d018c221a827b7edda4f013bfc919c1606\": rpc error: code = NotFound desc = could not find container \"fd43669314cad07c05d63f7aa69d80d018c221a827b7edda4f013bfc919c1606\": container with ID starting with fd43669314cad07c05d63f7aa69d80d018c221a827b7edda4f013bfc919c1606 not found: ID does not exist" Sep 11 18:31:12.695658 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:12.695632 2561 scope.go:117] "RemoveContainer" containerID="966aca739243da692077f59bc34833a01cdffbc4013aa25842441a919628b33d" Sep 11 18:31:12.695988 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:31:12.695956 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"966aca739243da692077f59bc34833a01cdffbc4013aa25842441a919628b33d\": container with ID starting with 966aca739243da692077f59bc34833a01cdffbc4013aa25842441a919628b33d not found: ID does not exist" containerID="966aca739243da692077f59bc34833a01cdffbc4013aa25842441a919628b33d" Sep 11 18:31:12.696067 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:12.695996 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"966aca739243da692077f59bc34833a01cdffbc4013aa25842441a919628b33d"} err="failed to get container status \"966aca739243da692077f59bc34833a01cdffbc4013aa25842441a919628b33d\": rpc error: code = NotFound desc = could not find container \"966aca739243da692077f59bc34833a01cdffbc4013aa25842441a919628b33d\": container with ID starting with 966aca739243da692077f59bc34833a01cdffbc4013aa25842441a919628b33d not found: ID does not exist" Sep 11 18:31:12.698208 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:12.698187 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-7e8ab9-openshift-default-74c8d54db-hlzzn"] Sep 11 18:31:13.139567 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:13.139486 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3730018e-ef6b-4210-9007-0861b85202df" path="/var/lib/kubelet/pods/3730018e-ef6b-4210-9007-0861b85202df/volumes" Sep 11 18:31:15.141296 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:15.141262 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c7c83f-6dcc6d86cc-mpw6w"] Sep 11 18:31:15.141774 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:15.141754 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3730018e-ef6b-4210-9007-0861b85202df" containerName="istio-proxy" Sep 11 18:31:15.141774 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:15.141775 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3730018e-ef6b-4210-9007-0861b85202df" containerName="istio-proxy" Sep 11 18:31:15.141889 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:15.141869 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3730018e-ef6b-4210-9007-0861b85202df" containerName="istio-proxy" Sep 11 18:31:15.141889 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:15.141877 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3730018e-ef6b-4210-9007-0861b85202df" containerName="istio-proxy" Sep 11 18:31:15.146836 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:15.146811 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-c7c83f-6dcc6d86cc-mpw6w" Sep 11 18:31:15.149145 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:15.149118 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-c7c83f\"" Sep 11 18:31:15.149498 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:15.149468 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-c7c83f-dockercfg-6lvqt\"" Sep 11 18:31:15.149842 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:15.149818 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c7c83f-6dcc6d86cc-mpw6w"] Sep 11 18:31:15.253756 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:15.253460 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2skmq\" (UniqueName: \"kubernetes.io/projected/76bf21cf-8897-4f9c-a22c-c7f437ff4d19-kube-api-access-2skmq\") pod \"payload-processing-e2e-explicit-c7c83f-6dcc6d86cc-mpw6w\" (UID: \"76bf21cf-8897-4f9c-a22c-c7f437ff4d19\") " pod="openshift-ingress/payload-processing-e2e-explicit-c7c83f-6dcc6d86cc-mpw6w" Sep 11 18:31:15.253756 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:15.253583 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76bf21cf-8897-4f9c-a22c-c7f437ff4d19-plugins-config-volume\") pod \"payload-processing-e2e-explicit-c7c83f-6dcc6d86cc-mpw6w\" (UID: \"76bf21cf-8897-4f9c-a22c-c7f437ff4d19\") " pod="openshift-ingress/payload-processing-e2e-explicit-c7c83f-6dcc6d86cc-mpw6w" Sep 11 18:31:15.354644 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:15.354587 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76bf21cf-8897-4f9c-a22c-c7f437ff4d19-plugins-config-volume\") pod \"payload-processing-e2e-explicit-c7c83f-6dcc6d86cc-mpw6w\" (UID: \"76bf21cf-8897-4f9c-a22c-c7f437ff4d19\") " pod="openshift-ingress/payload-processing-e2e-explicit-c7c83f-6dcc6d86cc-mpw6w" Sep 11 18:31:15.354830 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:15.354794 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2skmq\" (UniqueName: \"kubernetes.io/projected/76bf21cf-8897-4f9c-a22c-c7f437ff4d19-kube-api-access-2skmq\") pod \"payload-processing-e2e-explicit-c7c83f-6dcc6d86cc-mpw6w\" (UID: \"76bf21cf-8897-4f9c-a22c-c7f437ff4d19\") " pod="openshift-ingress/payload-processing-e2e-explicit-c7c83f-6dcc6d86cc-mpw6w" Sep 11 18:31:15.355391 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:15.355370 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76bf21cf-8897-4f9c-a22c-c7f437ff4d19-plugins-config-volume\") pod \"payload-processing-e2e-explicit-c7c83f-6dcc6d86cc-mpw6w\" (UID: \"76bf21cf-8897-4f9c-a22c-c7f437ff4d19\") " pod="openshift-ingress/payload-processing-e2e-explicit-c7c83f-6dcc6d86cc-mpw6w" Sep 11 18:31:15.364362 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:15.364331 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2skmq\" (UniqueName: \"kubernetes.io/projected/76bf21cf-8897-4f9c-a22c-c7f437ff4d19-kube-api-access-2skmq\") pod \"payload-processing-e2e-explicit-c7c83f-6dcc6d86cc-mpw6w\" (UID: \"76bf21cf-8897-4f9c-a22c-c7f437ff4d19\") " pod="openshift-ingress/payload-processing-e2e-explicit-c7c83f-6dcc6d86cc-mpw6w" Sep 11 18:31:15.460279 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:15.460234 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-c7c83f-6dcc6d86cc-mpw6w" Sep 11 18:31:15.739261 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:15.739225 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c7c83f-6dcc6d86cc-mpw6w"] Sep 11 18:31:15.742664 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:31:15.742543 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76bf21cf_8897_4f9c_a22c_c7f437ff4d19.slice/crio-e73dd404572aec125826144f38d1406569343eaf4b0f79dc960d1f3847798485 WatchSource:0}: Error finding container e73dd404572aec125826144f38d1406569343eaf4b0f79dc960d1f3847798485: Status 404 returned error can't find the container with id e73dd404572aec125826144f38d1406569343eaf4b0f79dc960d1f3847798485 Sep 11 18:31:16.697641 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:16.697605 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-c7c83f-6dcc6d86cc-mpw6w" event={"ID":"76bf21cf-8897-4f9c-a22c-c7f437ff4d19","Type":"ContainerStarted","Data":"65730be3b932651859cfff6a1c64870ce6acff2e18a3e9cbea6de5a5510272e6"} Sep 11 18:31:16.698031 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:16.697665 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-c7c83f-6dcc6d86cc-mpw6w" event={"ID":"76bf21cf-8897-4f9c-a22c-c7f437ff4d19","Type":"ContainerStarted","Data":"e73dd404572aec125826144f38d1406569343eaf4b0f79dc960d1f3847798485"} Sep 11 18:31:16.698031 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:16.697687 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-c7c83f-6dcc6d86cc-mpw6w" Sep 11 18:31:16.718940 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:16.718889 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-c7c83f-6dcc6d86cc-mpw6w" podStartSLOduration=1.7188732450000002 podStartE2EDuration="1.718873245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:31:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:31:16.716771355 +0000 UTC m=+958.213789113" watchObservedRunningTime="2026-09-11 18:31:16.718873245 +0000 UTC m=+958.215891038" Sep 11 18:31:27.707418 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:27.707386 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-c7c83f-6dcc6d86cc-mpw6w" Sep 11 18:31:36.848483 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:36.848447 2561 generic.go:356] "Generic (PLEG): container finished" podID="1ab46e7b-3161-46a7-9603-629d75d31c1d" containerID="d108db93c1153fe7bcaa3cacfa3d4c4298feeacdf22a149ef107297d56e356e6" exitCode=0 Sep 11 18:31:36.849006 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:36.848528 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm" event={"ID":"1ab46e7b-3161-46a7-9603-629d75d31c1d","Type":"ContainerDied","Data":"d108db93c1153fe7bcaa3cacfa3d4c4298feeacdf22a149ef107297d56e356e6"} Sep 11 18:31:36.849129 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:36.849108 2561 scope.go:117] "RemoveContainer" containerID="d108db93c1153fe7bcaa3cacfa3d4c4298feeacdf22a149ef107297d56e356e6" Sep 11 18:31:36.850376 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:36.850347 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:31:36.850458 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:36.850425 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:31:36.850496 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:36.850458 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:31:37.857527 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:37.857488 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm" event={"ID":"1ab46e7b-3161-46a7-9603-629d75d31c1d","Type":"ContainerStarted","Data":"dec4d2d366455dbae3f4090c9fa6df16dc504414765e7752ae8c704231c036a7"} Sep 11 18:31:37.862486 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:37.862458 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm" Sep 11 18:31:37.863788 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:37.863764 2561 patch_prober.go:28] interesting pod/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Sep 11 18:31:37.863911 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:37.863819 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm" podUID="1ab46e7b-3161-46a7-9603-629d75d31c1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Sep 11 18:31:38.862719 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:38.862686 2561 patch_prober.go:28] interesting pod/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Sep 11 18:31:38.863141 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:38.862743 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm" podUID="1ab46e7b-3161-46a7-9603-629d75d31c1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Sep 11 18:31:39.863775 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:39.863727 2561 patch_prober.go:28] interesting pod/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Sep 11 18:31:39.864262 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:39.863793 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm" podUID="1ab46e7b-3161-46a7-9603-629d75d31c1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Sep 11 18:31:39.900088 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:39.897885 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24"] Sep 11 18:31:39.900286 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:39.900125 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3730018e-ef6b-4210-9007-0861b85202df" containerName="istio-proxy" Sep 11 18:31:39.900286 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:39.900161 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3730018e-ef6b-4210-9007-0861b85202df" containerName="istio-proxy" Sep 11 18:31:39.920455 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:39.920258 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24"] Sep 11 18:31:39.920455 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:39.920386 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24" Sep 11 18:31:39.923374 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:39.923348 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-3ad8c202-openshift-default-dockercfg-zrrbg\"" Sep 11 18:31:40.033085 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:40.032365 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5rtv\" (UniqueName: \"kubernetes.io/projected/0fba78f9-b71f-4c15-82bd-ec5d60cf722a-kube-api-access-w5rtv\") pod \"e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24\" (UID: \"0fba78f9-b71f-4c15-82bd-ec5d60cf722a\") " pod="openshift-ingress/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24" Sep 11 18:31:40.033085 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:40.032438 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fba78f9-b71f-4c15-82bd-ec5d60cf722a-workload-certs\") pod \"e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24\" (UID: \"0fba78f9-b71f-4c15-82bd-ec5d60cf722a\") " pod="openshift-ingress/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24" Sep 11 18:31:40.033085 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:40.032466 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fba78f9-b71f-4c15-82bd-ec5d60cf722a-istio-envoy\") pod \"e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24\" (UID: \"0fba78f9-b71f-4c15-82bd-ec5d60cf722a\") " pod="openshift-ingress/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24" Sep 11 18:31:40.033085 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:40.032565 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fba78f9-b71f-4c15-82bd-ec5d60cf722a-istio-data\") pod \"e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24\" (UID: \"0fba78f9-b71f-4c15-82bd-ec5d60cf722a\") " pod="openshift-ingress/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24" Sep 11 18:31:40.037075 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:40.033800 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fba78f9-b71f-4c15-82bd-ec5d60cf722a-workload-socket\") pod \"e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24\" (UID: \"0fba78f9-b71f-4c15-82bd-ec5d60cf722a\") " pod="openshift-ingress/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24" Sep 11 18:31:40.037075 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:40.033895 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fba78f9-b71f-4c15-82bd-ec5d60cf722a-credential-socket\") pod \"e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24\" (UID: \"0fba78f9-b71f-4c15-82bd-ec5d60cf722a\") " pod="openshift-ingress/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24" Sep 11 18:31:40.037075 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:40.034092 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fba78f9-b71f-4c15-82bd-ec5d60cf722a-istio-token\") pod \"e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24\" (UID: \"0fba78f9-b71f-4c15-82bd-ec5d60cf722a\") " pod="openshift-ingress/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24" Sep 11 18:31:40.037075 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:40.034122 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fba78f9-b71f-4c15-82bd-ec5d60cf722a-istiod-ca-cert\") pod \"e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24\" (UID: \"0fba78f9-b71f-4c15-82bd-ec5d60cf722a\") " pod="openshift-ingress/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24" Sep 11 18:31:40.037075 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:40.034194 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fba78f9-b71f-4c15-82bd-ec5d60cf722a-istio-podinfo\") pod \"e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24\" (UID: \"0fba78f9-b71f-4c15-82bd-ec5d60cf722a\") " pod="openshift-ingress/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24" Sep 11 18:31:40.134827 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:40.134740 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fba78f9-b71f-4c15-82bd-ec5d60cf722a-workload-socket\") pod \"e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24\" (UID: \"0fba78f9-b71f-4c15-82bd-ec5d60cf722a\") " pod="openshift-ingress/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24" Sep 11 18:31:40.135025 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:40.135009 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fba78f9-b71f-4c15-82bd-ec5d60cf722a-credential-socket\") pod \"e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24\" (UID: \"0fba78f9-b71f-4c15-82bd-ec5d60cf722a\") " pod="openshift-ingress/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24" Sep 11 18:31:40.135177 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:40.135156 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fba78f9-b71f-4c15-82bd-ec5d60cf722a-workload-socket\") pod \"e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24\" (UID: \"0fba78f9-b71f-4c15-82bd-ec5d60cf722a\") " pod="openshift-ingress/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24" Sep 11 18:31:40.135257 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:40.135230 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fba78f9-b71f-4c15-82bd-ec5d60cf722a-istio-token\") pod \"e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24\" (UID: \"0fba78f9-b71f-4c15-82bd-ec5d60cf722a\") " pod="openshift-ingress/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24" Sep 11 18:31:40.135334 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:40.135278 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fba78f9-b71f-4c15-82bd-ec5d60cf722a-istiod-ca-cert\") pod \"e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24\" (UID: \"0fba78f9-b71f-4c15-82bd-ec5d60cf722a\") " pod="openshift-ingress/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24" Sep 11 18:31:40.135390 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:40.135332 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fba78f9-b71f-4c15-82bd-ec5d60cf722a-credential-socket\") pod \"e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24\" (UID: \"0fba78f9-b71f-4c15-82bd-ec5d60cf722a\") " pod="openshift-ingress/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24" Sep 11 18:31:40.135390 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:40.135336 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fba78f9-b71f-4c15-82bd-ec5d60cf722a-istio-podinfo\") pod \"e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24\" (UID: \"0fba78f9-b71f-4c15-82bd-ec5d60cf722a\") " pod="openshift-ingress/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24" Sep 11 18:31:40.135390 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:40.135380 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5rtv\" (UniqueName: \"kubernetes.io/projected/0fba78f9-b71f-4c15-82bd-ec5d60cf722a-kube-api-access-w5rtv\") pod \"e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24\" (UID: \"0fba78f9-b71f-4c15-82bd-ec5d60cf722a\") " pod="openshift-ingress/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24" Sep 11 18:31:40.135547 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:40.135411 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fba78f9-b71f-4c15-82bd-ec5d60cf722a-workload-certs\") pod \"e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24\" (UID: \"0fba78f9-b71f-4c15-82bd-ec5d60cf722a\") " pod="openshift-ingress/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24" Sep 11 18:31:40.135547 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:40.135446 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fba78f9-b71f-4c15-82bd-ec5d60cf722a-istio-envoy\") pod \"e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24\" (UID: \"0fba78f9-b71f-4c15-82bd-ec5d60cf722a\") " pod="openshift-ingress/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24" Sep 11 18:31:40.135547 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:40.135503 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fba78f9-b71f-4c15-82bd-ec5d60cf722a-istio-data\") pod \"e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24\" (UID: \"0fba78f9-b71f-4c15-82bd-ec5d60cf722a\") " pod="openshift-ingress/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24" Sep 11 18:31:40.135859 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:40.135835 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fba78f9-b71f-4c15-82bd-ec5d60cf722a-workload-certs\") pod \"e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24\" (UID: \"0fba78f9-b71f-4c15-82bd-ec5d60cf722a\") " pod="openshift-ingress/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24" Sep 11 18:31:40.136035 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:40.135843 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fba78f9-b71f-4c15-82bd-ec5d60cf722a-istio-data\") pod \"e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24\" (UID: \"0fba78f9-b71f-4c15-82bd-ec5d60cf722a\") " pod="openshift-ingress/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24" Sep 11 18:31:40.136180 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:40.136018 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fba78f9-b71f-4c15-82bd-ec5d60cf722a-istiod-ca-cert\") pod \"e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24\" (UID: \"0fba78f9-b71f-4c15-82bd-ec5d60cf722a\") " pod="openshift-ingress/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24" Sep 11 18:31:40.138394 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:40.138033 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fba78f9-b71f-4c15-82bd-ec5d60cf722a-istio-podinfo\") pod \"e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24\" (UID: \"0fba78f9-b71f-4c15-82bd-ec5d60cf722a\") " pod="openshift-ingress/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24" Sep 11 18:31:40.138394 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:40.138227 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fba78f9-b71f-4c15-82bd-ec5d60cf722a-istio-envoy\") pod \"e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24\" (UID: \"0fba78f9-b71f-4c15-82bd-ec5d60cf722a\") " pod="openshift-ingress/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24" Sep 11 18:31:40.148173 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:40.148148 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5rtv\" (UniqueName: \"kubernetes.io/projected/0fba78f9-b71f-4c15-82bd-ec5d60cf722a-kube-api-access-w5rtv\") pod \"e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24\" (UID: \"0fba78f9-b71f-4c15-82bd-ec5d60cf722a\") " pod="openshift-ingress/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24" Sep 11 18:31:40.148393 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:40.148371 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fba78f9-b71f-4c15-82bd-ec5d60cf722a-istio-token\") pod \"e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24\" (UID: \"0fba78f9-b71f-4c15-82bd-ec5d60cf722a\") " pod="openshift-ingress/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24" Sep 11 18:31:40.235431 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:40.235386 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24" Sep 11 18:31:40.551284 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:40.551137 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24"] Sep 11 18:31:40.551988 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:31:40.551943 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fba78f9_b71f_4c15_82bd_ec5d60cf722a.slice/crio-880a44654a50d008819d40558563b45d1f080cbb52c0dc44290d43d7f92858d7 WatchSource:0}: Error finding container 880a44654a50d008819d40558563b45d1f080cbb52c0dc44290d43d7f92858d7: Status 404 returned error can't find the container with id 880a44654a50d008819d40558563b45d1f080cbb52c0dc44290d43d7f92858d7 Sep 11 18:31:40.554307 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:40.554272 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:31:40.554395 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:40.554342 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:31:40.554395 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:40.554377 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:31:40.650314 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:40.650278 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm" Sep 11 18:31:40.862983 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:40.862886 2561 patch_prober.go:28] interesting pod/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Sep 11 18:31:40.862983 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:40.862943 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm" podUID="1ab46e7b-3161-46a7-9603-629d75d31c1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Sep 11 18:31:40.884963 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:40.884913 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24" event={"ID":"0fba78f9-b71f-4c15-82bd-ec5d60cf722a","Type":"ContainerStarted","Data":"55a1a1bedaa8a80f417488d7ba9a10fb185de5b2e68b4335e50be5d436e8dc00"} Sep 11 18:31:40.885457 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:40.884992 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24" event={"ID":"0fba78f9-b71f-4c15-82bd-ec5d60cf722a","Type":"ContainerStarted","Data":"880a44654a50d008819d40558563b45d1f080cbb52c0dc44290d43d7f92858d7"} Sep 11 18:31:40.912077 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:40.911994 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24" podStartSLOduration=1.911971844 podStartE2EDuration="1.911971844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:31:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:31:40.906912517 +0000 UTC m=+982.403930273" watchObservedRunningTime="2026-09-11 18:31:40.911971844 +0000 UTC m=+982.408989600" Sep 11 18:31:41.235736 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:41.235706 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24" Sep 11 18:31:41.240840 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:41.240811 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24" Sep 11 18:31:41.863381 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:41.863341 2561 patch_prober.go:28] interesting pod/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Sep 11 18:31:41.863555 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:41.863417 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm" podUID="1ab46e7b-3161-46a7-9603-629d75d31c1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Sep 11 18:31:41.898791 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:41.898748 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24" Sep 11 18:31:41.904739 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:41.904699 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24" Sep 11 18:31:42.863249 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:42.863209 2561 patch_prober.go:28] interesting pod/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Sep 11 18:31:42.863459 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:42.863285 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm" podUID="1ab46e7b-3161-46a7-9603-629d75d31c1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Sep 11 18:31:43.867241 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:43.867211 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm" Sep 11 18:31:43.868304 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:43.868281 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm" Sep 11 18:31:52.125840 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:52.125803 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-1de19d/test-model-1de19d-kserve-8774f9845-58nlj"] Sep 11 18:31:52.129890 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:52.129867 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1de19d/test-model-1de19d-kserve-8774f9845-58nlj" Sep 11 18:31:52.132803 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:52.132771 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-1de19d\"/\"default-dockercfg-9rpt2\"" Sep 11 18:31:52.132803 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:52.132791 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-1de19d\"/\"test-model-1de19d-kserve-self-signed-certs\"" Sep 11 18:31:52.133003 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:52.132771 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-1de19d\"/\"kube-root-ca.crt\"" Sep 11 18:31:52.133003 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:52.132779 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-1de19d\"/\"openshift-service-ca.crt\"" Sep 11 18:31:52.137465 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:52.137439 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-1de19d/test-model-1de19d-kserve-8774f9845-58nlj"] Sep 11 18:31:52.273012 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:52.272973 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e69c20c3-b07e-435d-adcd-1503fa87f3ce-tls-certs\") pod \"test-model-1de19d-kserve-8774f9845-58nlj\" (UID: \"e69c20c3-b07e-435d-adcd-1503fa87f3ce\") " pod="ai-tenant-e2e-inf-b-1de19d/test-model-1de19d-kserve-8774f9845-58nlj" Sep 11 18:31:52.273209 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:52.273024 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e69c20c3-b07e-435d-adcd-1503fa87f3ce-dshm\") pod \"test-model-1de19d-kserve-8774f9845-58nlj\" (UID: \"e69c20c3-b07e-435d-adcd-1503fa87f3ce\") " pod="ai-tenant-e2e-inf-b-1de19d/test-model-1de19d-kserve-8774f9845-58nlj" Sep 11 18:31:52.273209 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:52.273050 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e69c20c3-b07e-435d-adcd-1503fa87f3ce-model-cache\") pod \"test-model-1de19d-kserve-8774f9845-58nlj\" (UID: \"e69c20c3-b07e-435d-adcd-1503fa87f3ce\") " pod="ai-tenant-e2e-inf-b-1de19d/test-model-1de19d-kserve-8774f9845-58nlj" Sep 11 18:31:52.273209 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:52.273098 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e69c20c3-b07e-435d-adcd-1503fa87f3ce-tmp-dir\") pod \"test-model-1de19d-kserve-8774f9845-58nlj\" (UID: \"e69c20c3-b07e-435d-adcd-1503fa87f3ce\") " pod="ai-tenant-e2e-inf-b-1de19d/test-model-1de19d-kserve-8774f9845-58nlj" Sep 11 18:31:52.273209 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:52.273155 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbg4d\" (UniqueName: \"kubernetes.io/projected/e69c20c3-b07e-435d-adcd-1503fa87f3ce-kube-api-access-mbg4d\") pod \"test-model-1de19d-kserve-8774f9845-58nlj\" (UID: \"e69c20c3-b07e-435d-adcd-1503fa87f3ce\") " pod="ai-tenant-e2e-inf-b-1de19d/test-model-1de19d-kserve-8774f9845-58nlj" Sep 11 18:31:52.273352 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:52.273230 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e69c20c3-b07e-435d-adcd-1503fa87f3ce-home\") pod \"test-model-1de19d-kserve-8774f9845-58nlj\" (UID: \"e69c20c3-b07e-435d-adcd-1503fa87f3ce\") " pod="ai-tenant-e2e-inf-b-1de19d/test-model-1de19d-kserve-8774f9845-58nlj" Sep 11 18:31:52.374597 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:52.374559 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e69c20c3-b07e-435d-adcd-1503fa87f3ce-tls-certs\") pod \"test-model-1de19d-kserve-8774f9845-58nlj\" (UID: \"e69c20c3-b07e-435d-adcd-1503fa87f3ce\") " pod="ai-tenant-e2e-inf-b-1de19d/test-model-1de19d-kserve-8774f9845-58nlj" Sep 11 18:31:52.374784 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:52.374605 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e69c20c3-b07e-435d-adcd-1503fa87f3ce-dshm\") pod \"test-model-1de19d-kserve-8774f9845-58nlj\" (UID: \"e69c20c3-b07e-435d-adcd-1503fa87f3ce\") " pod="ai-tenant-e2e-inf-b-1de19d/test-model-1de19d-kserve-8774f9845-58nlj" Sep 11 18:31:52.374784 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:52.374642 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e69c20c3-b07e-435d-adcd-1503fa87f3ce-model-cache\") pod \"test-model-1de19d-kserve-8774f9845-58nlj\" (UID: \"e69c20c3-b07e-435d-adcd-1503fa87f3ce\") " pod="ai-tenant-e2e-inf-b-1de19d/test-model-1de19d-kserve-8774f9845-58nlj" Sep 11 18:31:52.374784 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:52.374664 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e69c20c3-b07e-435d-adcd-1503fa87f3ce-tmp-dir\") pod \"test-model-1de19d-kserve-8774f9845-58nlj\" (UID: \"e69c20c3-b07e-435d-adcd-1503fa87f3ce\") " pod="ai-tenant-e2e-inf-b-1de19d/test-model-1de19d-kserve-8774f9845-58nlj" Sep 11 18:31:52.374784 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:52.374712 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbg4d\" (UniqueName: \"kubernetes.io/projected/e69c20c3-b07e-435d-adcd-1503fa87f3ce-kube-api-access-mbg4d\") pod \"test-model-1de19d-kserve-8774f9845-58nlj\" (UID: \"e69c20c3-b07e-435d-adcd-1503fa87f3ce\") " pod="ai-tenant-e2e-inf-b-1de19d/test-model-1de19d-kserve-8774f9845-58nlj" Sep 11 18:31:52.374989 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:52.374802 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e69c20c3-b07e-435d-adcd-1503fa87f3ce-home\") pod \"test-model-1de19d-kserve-8774f9845-58nlj\" (UID: \"e69c20c3-b07e-435d-adcd-1503fa87f3ce\") " pod="ai-tenant-e2e-inf-b-1de19d/test-model-1de19d-kserve-8774f9845-58nlj" Sep 11 18:31:52.375178 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:52.375147 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e69c20c3-b07e-435d-adcd-1503fa87f3ce-tmp-dir\") pod \"test-model-1de19d-kserve-8774f9845-58nlj\" (UID: \"e69c20c3-b07e-435d-adcd-1503fa87f3ce\") " pod="ai-tenant-e2e-inf-b-1de19d/test-model-1de19d-kserve-8774f9845-58nlj" Sep 11 18:31:52.375312 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:52.375231 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e69c20c3-b07e-435d-adcd-1503fa87f3ce-home\") pod \"test-model-1de19d-kserve-8774f9845-58nlj\" (UID: \"e69c20c3-b07e-435d-adcd-1503fa87f3ce\") " pod="ai-tenant-e2e-inf-b-1de19d/test-model-1de19d-kserve-8774f9845-58nlj" Sep 11 18:31:52.375312 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:52.375270 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e69c20c3-b07e-435d-adcd-1503fa87f3ce-model-cache\") pod \"test-model-1de19d-kserve-8774f9845-58nlj\" (UID: \"e69c20c3-b07e-435d-adcd-1503fa87f3ce\") " pod="ai-tenant-e2e-inf-b-1de19d/test-model-1de19d-kserve-8774f9845-58nlj" Sep 11 18:31:52.377139 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:52.377083 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e69c20c3-b07e-435d-adcd-1503fa87f3ce-dshm\") pod \"test-model-1de19d-kserve-8774f9845-58nlj\" (UID: \"e69c20c3-b07e-435d-adcd-1503fa87f3ce\") " pod="ai-tenant-e2e-inf-b-1de19d/test-model-1de19d-kserve-8774f9845-58nlj" Sep 11 18:31:52.377235 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:52.377197 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e69c20c3-b07e-435d-adcd-1503fa87f3ce-tls-certs\") pod \"test-model-1de19d-kserve-8774f9845-58nlj\" (UID: \"e69c20c3-b07e-435d-adcd-1503fa87f3ce\") " pod="ai-tenant-e2e-inf-b-1de19d/test-model-1de19d-kserve-8774f9845-58nlj" Sep 11 18:31:52.383849 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:52.383824 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbg4d\" (UniqueName: \"kubernetes.io/projected/e69c20c3-b07e-435d-adcd-1503fa87f3ce-kube-api-access-mbg4d\") pod \"test-model-1de19d-kserve-8774f9845-58nlj\" (UID: \"e69c20c3-b07e-435d-adcd-1503fa87f3ce\") " pod="ai-tenant-e2e-inf-b-1de19d/test-model-1de19d-kserve-8774f9845-58nlj" Sep 11 18:31:52.442345 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:52.442306 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1de19d/test-model-1de19d-kserve-8774f9845-58nlj" Sep 11 18:31:52.727105 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:52.727071 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-1de19d/test-model-1de19d-kserve-8774f9845-58nlj"] Sep 11 18:31:52.728193 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:31:52.728158 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode69c20c3_b07e_435d_adcd_1503fa87f3ce.slice/crio-2d04f348325780752db5f480a098a1003ca572df9ae0a8b0f26c0036eb0ac6b3 WatchSource:0}: Error finding container 2d04f348325780752db5f480a098a1003ca572df9ae0a8b0f26c0036eb0ac6b3: Status 404 returned error can't find the container with id 2d04f348325780752db5f480a098a1003ca572df9ae0a8b0f26c0036eb0ac6b3 Sep 11 18:31:52.978330 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:52.978241 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1de19d/test-model-1de19d-kserve-8774f9845-58nlj" event={"ID":"e69c20c3-b07e-435d-adcd-1503fa87f3ce","Type":"ContainerStarted","Data":"ca51057024ab02d24dedd9d65f821d660e39931bee3d8a0699d0d71372437380"} Sep 11 18:31:52.978330 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:52.978293 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1de19d/test-model-1de19d-kserve-8774f9845-58nlj" event={"ID":"e69c20c3-b07e-435d-adcd-1503fa87f3ce","Type":"ContainerStarted","Data":"2d04f348325780752db5f480a098a1003ca572df9ae0a8b0f26c0036eb0ac6b3"} Sep 11 18:31:53.006660 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:31:53.006603 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-1de19d/test-model-1de19d-kserve-8774f9845-58nlj" podStartSLOduration=1.006583531 podStartE2EDuration="1.006583531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:31:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:31:53.004540765 +0000 UTC m=+994.501558513" watchObservedRunningTime="2026-09-11 18:31:53.006583531 +0000 UTC m=+994.503601285" Sep 11 18:32:02.442990 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:02.442936 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-1de19d/test-model-1de19d-kserve-8774f9845-58nlj" Sep 11 18:32:02.443478 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:02.443015 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-1de19d/test-model-1de19d-kserve-8774f9845-58nlj" Sep 11 18:32:02.457660 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:02.457628 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-1de19d/test-model-1de19d-kserve-8774f9845-58nlj" Sep 11 18:32:03.066933 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:03.066902 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-1de19d/test-model-1de19d-kserve-8774f9845-58nlj" Sep 11 18:32:12.942237 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:12.942184 2561 patch_prober.go:28] interesting pod/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 11 18:32:12.942885 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:12.942260 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24" podUID="0fba78f9-b71f-4c15-82bd-ec5d60cf722a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 11 18:32:14.145257 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:14.145214 2561 generic.go:356] "Generic (PLEG): container finished" podID="0fba78f9-b71f-4c15-82bd-ec5d60cf722a" containerID="55a1a1bedaa8a80f417488d7ba9a10fb185de5b2e68b4335e50be5d436e8dc00" exitCode=0 Sep 11 18:32:14.145741 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:14.145287 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24" event={"ID":"0fba78f9-b71f-4c15-82bd-ec5d60cf722a","Type":"ContainerDied","Data":"55a1a1bedaa8a80f417488d7ba9a10fb185de5b2e68b4335e50be5d436e8dc00"} Sep 11 18:32:14.145741 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:14.145690 2561 scope.go:117] "RemoveContainer" containerID="55a1a1bedaa8a80f417488d7ba9a10fb185de5b2e68b4335e50be5d436e8dc00" Sep 11 18:32:14.146926 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:14.146892 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:32:14.147040 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:14.146982 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:32:14.147149 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:14.147035 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:32:15.153864 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:15.153827 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24" event={"ID":"0fba78f9-b71f-4c15-82bd-ec5d60cf722a","Type":"ContainerStarted","Data":"37419bfc5b0fc19039fdad6ce4cfa598a52b852c989988afbc7239b1f8af71e1"} Sep 11 18:32:15.236639 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:15.236601 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24" Sep 11 18:32:15.236867 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:15.236836 2561 patch_prober.go:28] interesting pod/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Sep 11 18:32:15.236952 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:15.236900 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24" podUID="0fba78f9-b71f-4c15-82bd-ec5d60cf722a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Sep 11 18:32:16.236681 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:16.236642 2561 patch_prober.go:28] interesting pod/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Sep 11 18:32:16.237199 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:16.236723 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24" podUID="0fba78f9-b71f-4c15-82bd-ec5d60cf722a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Sep 11 18:32:17.235843 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:17.235796 2561 patch_prober.go:28] interesting pod/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Sep 11 18:32:17.236075 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:17.235886 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24" podUID="0fba78f9-b71f-4c15-82bd-ec5d60cf722a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Sep 11 18:32:18.236646 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:18.236600 2561 patch_prober.go:28] interesting pod/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Sep 11 18:32:18.237237 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:18.236673 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24" podUID="0fba78f9-b71f-4c15-82bd-ec5d60cf722a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Sep 11 18:32:19.236344 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:19.236308 2561 patch_prober.go:28] interesting pod/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Sep 11 18:32:19.236548 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:19.236373 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24" podUID="0fba78f9-b71f-4c15-82bd-ec5d60cf722a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Sep 11 18:32:20.235961 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:20.235926 2561 patch_prober.go:28] interesting pod/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Sep 11 18:32:20.236375 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:20.235982 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24" podUID="0fba78f9-b71f-4c15-82bd-ec5d60cf722a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Sep 11 18:32:21.239773 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:21.239742 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24" Sep 11 18:32:21.240261 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:21.240247 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24" Sep 11 18:32:21.240969 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:21.240944 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24" Sep 11 18:32:23.209850 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:23.209809 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3ad8c202-nsrx4"] Sep 11 18:32:23.213560 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:23.213537 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3ad8c202-nsrx4" Sep 11 18:32:23.215744 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:23.215722 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n968b\"" Sep 11 18:32:23.215744 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:23.215735 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 18:32:23.223403 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:23.223374 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3ad8c202-nsrx4"] Sep 11 18:32:23.229640 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:23.229612 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b6dcaae-cac1-4be5-9465-e7d0817f2970-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-3ad8c202-nsrx4\" (UID: \"8b6dcaae-cac1-4be5-9465-e7d0817f2970\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3ad8c202-nsrx4" Sep 11 18:32:23.331140 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:23.331101 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b6dcaae-cac1-4be5-9465-e7d0817f2970-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-3ad8c202-nsrx4\" (UID: \"8b6dcaae-cac1-4be5-9465-e7d0817f2970\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3ad8c202-nsrx4" Sep 11 18:32:23.331758 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:23.331740 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b6dcaae-cac1-4be5-9465-e7d0817f2970-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-3ad8c202-nsrx4\" (UID: \"8b6dcaae-cac1-4be5-9465-e7d0817f2970\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3ad8c202-nsrx4" Sep 11 18:32:23.527985 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:23.527893 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3ad8c202-nsrx4" Sep 11 18:32:23.809270 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:23.809235 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3ad8c202-nsrx4"] Sep 11 18:32:23.810520 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:32:23.810494 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b6dcaae_cac1_4be5_9465_e7d0817f2970.slice/crio-029878f3b5b450dbf6c43a6743414f97f50aa401b0b25fc24f67e0d5415e07a5 WatchSource:0}: Error finding container 029878f3b5b450dbf6c43a6743414f97f50aa401b0b25fc24f67e0d5415e07a5: Status 404 returned error can't find the container with id 029878f3b5b450dbf6c43a6743414f97f50aa401b0b25fc24f67e0d5415e07a5 Sep 11 18:32:24.223831 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:24.223797 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3ad8c202-nsrx4" event={"ID":"8b6dcaae-cac1-4be5-9465-e7d0817f2970","Type":"ContainerStarted","Data":"88025ca099ac755cd0847912032990f9d2b03b80d9007f76f3bb3bdd3072d374"} Sep 11 18:32:24.224272 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:24.223850 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3ad8c202-nsrx4" event={"ID":"8b6dcaae-cac1-4be5-9465-e7d0817f2970","Type":"ContainerStarted","Data":"029878f3b5b450dbf6c43a6743414f97f50aa401b0b25fc24f67e0d5415e07a5"} Sep 11 18:32:24.240637 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:24.240582 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3ad8c202-nsrx4" podStartSLOduration=1.240538489 podStartE2EDuration="1.240538489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:32:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:32:24.239113799 +0000 UTC m=+1025.736131552" watchObservedRunningTime="2026-09-11 18:32:24.240538489 +0000 UTC m=+1025.737556243" Sep 11 18:32:25.232506 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:25.232471 2561 generic.go:356] "Generic (PLEG): container finished" podID="8b6dcaae-cac1-4be5-9465-e7d0817f2970" containerID="88025ca099ac755cd0847912032990f9d2b03b80d9007f76f3bb3bdd3072d374" exitCode=0 Sep 11 18:32:25.232913 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:25.232560 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3ad8c202-nsrx4" event={"ID":"8b6dcaae-cac1-4be5-9465-e7d0817f2970","Type":"ContainerDied","Data":"88025ca099ac755cd0847912032990f9d2b03b80d9007f76f3bb3bdd3072d374"} Sep 11 18:32:26.377354 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:26.377325 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3ad8c202-nsrx4" Sep 11 18:32:26.468289 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:26.468243 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b6dcaae-cac1-4be5-9465-e7d0817f2970-maas-api-service-ca\") pod \"8b6dcaae-cac1-4be5-9465-e7d0817f2970\" (UID: \"8b6dcaae-cac1-4be5-9465-e7d0817f2970\") " Sep 11 18:32:26.468608 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:26.468583 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b6dcaae-cac1-4be5-9465-e7d0817f2970-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8b6dcaae-cac1-4be5-9465-e7d0817f2970" (UID: "8b6dcaae-cac1-4be5-9465-e7d0817f2970"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:32:26.468749 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:26.468733 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b6dcaae-cac1-4be5-9465-e7d0817f2970-maas-api-service-ca\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:32:27.247430 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:27.247396 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3ad8c202-nsrx4" Sep 11 18:32:27.247631 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:27.247425 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3ad8c202-nsrx4" event={"ID":"8b6dcaae-cac1-4be5-9465-e7d0817f2970","Type":"ContainerDied","Data":"029878f3b5b450dbf6c43a6743414f97f50aa401b0b25fc24f67e0d5415e07a5"} Sep 11 18:32:27.247631 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:27.247495 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="029878f3b5b450dbf6c43a6743414f97f50aa401b0b25fc24f67e0d5415e07a5" Sep 11 18:32:33.974197 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:33.974152 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3ad8c202-nsrx4"] Sep 11 18:32:33.980807 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:33.980739 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3ad8c202-nsrx4"] Sep 11 18:32:35.140122 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:35.140063 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b6dcaae-cac1-4be5-9465-e7d0817f2970" path="/var/lib/kubelet/pods/8b6dcaae-cac1-4be5-9465-e7d0817f2970/volumes" Sep 11 18:32:41.151831 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:41.151798 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24"] Sep 11 18:32:41.152286 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:41.152044 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24" podUID="0fba78f9-b71f-4c15-82bd-ec5d60cf722a" containerName="istio-proxy" containerID="cri-o://37419bfc5b0fc19039fdad6ce4cfa598a52b852c989988afbc7239b1f8af71e1" gracePeriod=30 Sep 11 18:32:46.380579 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:46.380539 2561 generic.go:356] "Generic (PLEG): container finished" podID="0fba78f9-b71f-4c15-82bd-ec5d60cf722a" containerID="37419bfc5b0fc19039fdad6ce4cfa598a52b852c989988afbc7239b1f8af71e1" exitCode=0 Sep 11 18:32:46.380995 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:46.380612 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24" event={"ID":"0fba78f9-b71f-4c15-82bd-ec5d60cf722a","Type":"ContainerDied","Data":"37419bfc5b0fc19039fdad6ce4cfa598a52b852c989988afbc7239b1f8af71e1"} Sep 11 18:32:46.380995 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:46.380677 2561 scope.go:117] "RemoveContainer" containerID="55a1a1bedaa8a80f417488d7ba9a10fb185de5b2e68b4335e50be5d436e8dc00" Sep 11 18:32:46.425196 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:46.425136 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24" Sep 11 18:32:46.592630 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:46.592595 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fba78f9-b71f-4c15-82bd-ec5d60cf722a-istio-token\") pod \"0fba78f9-b71f-4c15-82bd-ec5d60cf722a\" (UID: \"0fba78f9-b71f-4c15-82bd-ec5d60cf722a\") " Sep 11 18:32:46.592841 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:46.592681 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fba78f9-b71f-4c15-82bd-ec5d60cf722a-workload-certs\") pod \"0fba78f9-b71f-4c15-82bd-ec5d60cf722a\" (UID: \"0fba78f9-b71f-4c15-82bd-ec5d60cf722a\") " Sep 11 18:32:46.592841 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:46.592720 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fba78f9-b71f-4c15-82bd-ec5d60cf722a-istio-data\") pod \"0fba78f9-b71f-4c15-82bd-ec5d60cf722a\" (UID: \"0fba78f9-b71f-4c15-82bd-ec5d60cf722a\") " Sep 11 18:32:46.592841 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:46.592746 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fba78f9-b71f-4c15-82bd-ec5d60cf722a-istio-podinfo\") pod \"0fba78f9-b71f-4c15-82bd-ec5d60cf722a\" (UID: \"0fba78f9-b71f-4c15-82bd-ec5d60cf722a\") " Sep 11 18:32:46.592841 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:46.592778 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w5rtv\" (UniqueName: \"kubernetes.io/projected/0fba78f9-b71f-4c15-82bd-ec5d60cf722a-kube-api-access-w5rtv\") pod \"0fba78f9-b71f-4c15-82bd-ec5d60cf722a\" (UID: \"0fba78f9-b71f-4c15-82bd-ec5d60cf722a\") " Sep 11 18:32:46.593099 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:46.592938 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fba78f9-b71f-4c15-82bd-ec5d60cf722a-workload-socket\") pod \"0fba78f9-b71f-4c15-82bd-ec5d60cf722a\" (UID: \"0fba78f9-b71f-4c15-82bd-ec5d60cf722a\") " Sep 11 18:32:46.593099 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:46.592975 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fba78f9-b71f-4c15-82bd-ec5d60cf722a-istiod-ca-cert\") pod \"0fba78f9-b71f-4c15-82bd-ec5d60cf722a\" (UID: \"0fba78f9-b71f-4c15-82bd-ec5d60cf722a\") " Sep 11 18:32:46.593099 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:46.593014 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fba78f9-b71f-4c15-82bd-ec5d60cf722a-credential-socket\") pod \"0fba78f9-b71f-4c15-82bd-ec5d60cf722a\" (UID: \"0fba78f9-b71f-4c15-82bd-ec5d60cf722a\") " Sep 11 18:32:46.593099 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:46.593007 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fba78f9-b71f-4c15-82bd-ec5d60cf722a-istio-data" (OuterVolumeSpecName: "istio-data") pod "0fba78f9-b71f-4c15-82bd-ec5d60cf722a" (UID: "0fba78f9-b71f-4c15-82bd-ec5d60cf722a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:32:46.593322 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:46.593107 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fba78f9-b71f-4c15-82bd-ec5d60cf722a-istio-envoy\") pod \"0fba78f9-b71f-4c15-82bd-ec5d60cf722a\" (UID: \"0fba78f9-b71f-4c15-82bd-ec5d60cf722a\") " Sep 11 18:32:46.593322 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:46.593020 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fba78f9-b71f-4c15-82bd-ec5d60cf722a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0fba78f9-b71f-4c15-82bd-ec5d60cf722a" (UID: "0fba78f9-b71f-4c15-82bd-ec5d60cf722a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:32:46.593322 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:46.593239 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fba78f9-b71f-4c15-82bd-ec5d60cf722a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0fba78f9-b71f-4c15-82bd-ec5d60cf722a" (UID: "0fba78f9-b71f-4c15-82bd-ec5d60cf722a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:32:46.593322 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:46.593277 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fba78f9-b71f-4c15-82bd-ec5d60cf722a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0fba78f9-b71f-4c15-82bd-ec5d60cf722a" (UID: "0fba78f9-b71f-4c15-82bd-ec5d60cf722a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:32:46.593322 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:46.593312 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fba78f9-b71f-4c15-82bd-ec5d60cf722a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0fba78f9-b71f-4c15-82bd-ec5d60cf722a" (UID: "0fba78f9-b71f-4c15-82bd-ec5d60cf722a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:32:46.593573 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:46.593538 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fba78f9-b71f-4c15-82bd-ec5d60cf722a-workload-certs\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:32:46.593573 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:46.593556 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fba78f9-b71f-4c15-82bd-ec5d60cf722a-istio-data\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:32:46.593573 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:46.593570 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fba78f9-b71f-4c15-82bd-ec5d60cf722a-workload-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:32:46.593727 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:46.593583 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fba78f9-b71f-4c15-82bd-ec5d60cf722a-istiod-ca-cert\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:32:46.593727 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:46.593596 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fba78f9-b71f-4c15-82bd-ec5d60cf722a-credential-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:32:46.595160 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:46.595126 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fba78f9-b71f-4c15-82bd-ec5d60cf722a-kube-api-access-w5rtv" (OuterVolumeSpecName: "kube-api-access-w5rtv") pod "0fba78f9-b71f-4c15-82bd-ec5d60cf722a" (UID: "0fba78f9-b71f-4c15-82bd-ec5d60cf722a"). InnerVolumeSpecName "kube-api-access-w5rtv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:32:46.595312 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:46.595217 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fba78f9-b71f-4c15-82bd-ec5d60cf722a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0fba78f9-b71f-4c15-82bd-ec5d60cf722a" (UID: "0fba78f9-b71f-4c15-82bd-ec5d60cf722a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:32:46.595575 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:46.595549 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0fba78f9-b71f-4c15-82bd-ec5d60cf722a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0fba78f9-b71f-4c15-82bd-ec5d60cf722a" (UID: "0fba78f9-b71f-4c15-82bd-ec5d60cf722a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 18:32:46.595575 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:46.595560 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fba78f9-b71f-4c15-82bd-ec5d60cf722a-istio-token" (OuterVolumeSpecName: "istio-token") pod "0fba78f9-b71f-4c15-82bd-ec5d60cf722a" (UID: "0fba78f9-b71f-4c15-82bd-ec5d60cf722a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:32:46.694799 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:46.694696 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fba78f9-b71f-4c15-82bd-ec5d60cf722a-istio-podinfo\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:32:46.694799 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:46.694734 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5rtv\" (UniqueName: \"kubernetes.io/projected/0fba78f9-b71f-4c15-82bd-ec5d60cf722a-kube-api-access-w5rtv\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:32:46.694799 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:46.694750 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fba78f9-b71f-4c15-82bd-ec5d60cf722a-istio-envoy\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:32:46.694799 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:46.694764 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fba78f9-b71f-4c15-82bd-ec5d60cf722a-istio-token\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:32:47.390541 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:47.390504 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24" Sep 11 18:32:47.390541 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:47.390535 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24" event={"ID":"0fba78f9-b71f-4c15-82bd-ec5d60cf722a","Type":"ContainerDied","Data":"880a44654a50d008819d40558563b45d1f080cbb52c0dc44290d43d7f92858d7"} Sep 11 18:32:47.391118 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:47.390592 2561 scope.go:117] "RemoveContainer" containerID="37419bfc5b0fc19039fdad6ce4cfa598a52b852c989988afbc7239b1f8af71e1" Sep 11 18:32:47.411153 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:47.411019 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24"] Sep 11 18:32:47.416689 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:47.416657 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-3ad8c202-openshift-default-6f78799bbd-b8g24"] Sep 11 18:32:49.140386 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:49.140353 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fba78f9-b71f-4c15-82bd-ec5d60cf722a" path="/var/lib/kubelet/pods/0fba78f9-b71f-4c15-82bd-ec5d60cf722a/volumes" Sep 11 18:32:56.073466 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:56.073432 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t"] Sep 11 18:32:56.073882 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:56.073868 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b6dcaae-cac1-4be5-9465-e7d0817f2970" containerName="revoke-keys" Sep 11 18:32:56.073882 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:56.073883 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b6dcaae-cac1-4be5-9465-e7d0817f2970" containerName="revoke-keys" Sep 11 18:32:56.073973 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:56.073903 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fba78f9-b71f-4c15-82bd-ec5d60cf722a" containerName="istio-proxy" Sep 11 18:32:56.073973 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:56.073909 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fba78f9-b71f-4c15-82bd-ec5d60cf722a" containerName="istio-proxy" Sep 11 18:32:56.073973 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:56.073919 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fba78f9-b71f-4c15-82bd-ec5d60cf722a" containerName="istio-proxy" Sep 11 18:32:56.073973 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:56.073926 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fba78f9-b71f-4c15-82bd-ec5d60cf722a" containerName="istio-proxy" Sep 11 18:32:56.074137 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:56.073991 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fba78f9-b71f-4c15-82bd-ec5d60cf722a" containerName="istio-proxy" Sep 11 18:32:56.074137 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:56.074000 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fba78f9-b71f-4c15-82bd-ec5d60cf722a" containerName="istio-proxy" Sep 11 18:32:56.074137 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:56.074010 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b6dcaae-cac1-4be5-9465-e7d0817f2970" containerName="revoke-keys" Sep 11 18:32:56.076969 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:56.076948 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t" Sep 11 18:32:56.079246 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:56.079220 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-869176-openshift-default-dockercfg-lnxs4\"" Sep 11 18:32:56.091347 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:56.091317 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t"] Sep 11 18:32:56.200809 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:56.200685 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ecbed32-91f0-44b6-aa50-2ed6f61bdc98-istio-envoy\") pod \"e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t\" (UID: \"3ecbed32-91f0-44b6-aa50-2ed6f61bdc98\") " pod="openshift-ingress/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t" Sep 11 18:32:56.201019 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:56.200893 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94sd2\" (UniqueName: \"kubernetes.io/projected/3ecbed32-91f0-44b6-aa50-2ed6f61bdc98-kube-api-access-94sd2\") pod \"e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t\" (UID: \"3ecbed32-91f0-44b6-aa50-2ed6f61bdc98\") " pod="openshift-ingress/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t" Sep 11 18:32:56.201019 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:56.200929 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ecbed32-91f0-44b6-aa50-2ed6f61bdc98-workload-certs\") pod \"e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t\" (UID: \"3ecbed32-91f0-44b6-aa50-2ed6f61bdc98\") " pod="openshift-ingress/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t" Sep 11 18:32:56.201158 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:56.201018 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ecbed32-91f0-44b6-aa50-2ed6f61bdc98-credential-socket\") pod \"e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t\" (UID: \"3ecbed32-91f0-44b6-aa50-2ed6f61bdc98\") " pod="openshift-ingress/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t" Sep 11 18:32:56.201158 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:56.201095 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ecbed32-91f0-44b6-aa50-2ed6f61bdc98-istio-podinfo\") pod \"e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t\" (UID: \"3ecbed32-91f0-44b6-aa50-2ed6f61bdc98\") " pod="openshift-ingress/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t" Sep 11 18:32:56.201158 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:56.201124 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ecbed32-91f0-44b6-aa50-2ed6f61bdc98-istiod-ca-cert\") pod \"e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t\" (UID: \"3ecbed32-91f0-44b6-aa50-2ed6f61bdc98\") " pod="openshift-ingress/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t" Sep 11 18:32:56.201158 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:56.201155 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ecbed32-91f0-44b6-aa50-2ed6f61bdc98-istio-data\") pod \"e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t\" (UID: \"3ecbed32-91f0-44b6-aa50-2ed6f61bdc98\") " pod="openshift-ingress/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t" Sep 11 18:32:56.201369 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:56.201216 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ecbed32-91f0-44b6-aa50-2ed6f61bdc98-workload-socket\") pod \"e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t\" (UID: \"3ecbed32-91f0-44b6-aa50-2ed6f61bdc98\") " pod="openshift-ingress/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t" Sep 11 18:32:56.201369 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:56.201238 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ecbed32-91f0-44b6-aa50-2ed6f61bdc98-istio-token\") pod \"e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t\" (UID: \"3ecbed32-91f0-44b6-aa50-2ed6f61bdc98\") " pod="openshift-ingress/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t" Sep 11 18:32:56.302130 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:56.302079 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ecbed32-91f0-44b6-aa50-2ed6f61bdc98-istio-envoy\") pod \"e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t\" (UID: \"3ecbed32-91f0-44b6-aa50-2ed6f61bdc98\") " pod="openshift-ingress/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t" Sep 11 18:32:56.302351 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:56.302150 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94sd2\" (UniqueName: \"kubernetes.io/projected/3ecbed32-91f0-44b6-aa50-2ed6f61bdc98-kube-api-access-94sd2\") pod \"e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t\" (UID: \"3ecbed32-91f0-44b6-aa50-2ed6f61bdc98\") " pod="openshift-ingress/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t" Sep 11 18:32:56.302351 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:56.302178 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ecbed32-91f0-44b6-aa50-2ed6f61bdc98-workload-certs\") pod \"e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t\" (UID: \"3ecbed32-91f0-44b6-aa50-2ed6f61bdc98\") " pod="openshift-ingress/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t" Sep 11 18:32:56.302351 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:56.302228 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ecbed32-91f0-44b6-aa50-2ed6f61bdc98-credential-socket\") pod \"e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t\" (UID: \"3ecbed32-91f0-44b6-aa50-2ed6f61bdc98\") " pod="openshift-ingress/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t" Sep 11 18:32:56.302520 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:56.302444 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ecbed32-91f0-44b6-aa50-2ed6f61bdc98-istio-podinfo\") pod \"e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t\" (UID: \"3ecbed32-91f0-44b6-aa50-2ed6f61bdc98\") " pod="openshift-ingress/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t" Sep 11 18:32:56.302520 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:56.302499 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ecbed32-91f0-44b6-aa50-2ed6f61bdc98-istiod-ca-cert\") pod \"e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t\" (UID: \"3ecbed32-91f0-44b6-aa50-2ed6f61bdc98\") " pod="openshift-ingress/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t" Sep 11 18:32:56.302691 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:56.302534 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ecbed32-91f0-44b6-aa50-2ed6f61bdc98-istio-data\") pod \"e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t\" (UID: \"3ecbed32-91f0-44b6-aa50-2ed6f61bdc98\") " pod="openshift-ingress/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t" Sep 11 18:32:56.302691 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:56.302602 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ecbed32-91f0-44b6-aa50-2ed6f61bdc98-workload-certs\") pod \"e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t\" (UID: \"3ecbed32-91f0-44b6-aa50-2ed6f61bdc98\") " pod="openshift-ingress/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t" Sep 11 18:32:56.302691 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:56.302601 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ecbed32-91f0-44b6-aa50-2ed6f61bdc98-workload-socket\") pod \"e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t\" (UID: \"3ecbed32-91f0-44b6-aa50-2ed6f61bdc98\") " pod="openshift-ingress/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t" Sep 11 18:32:56.302691 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:56.302651 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ecbed32-91f0-44b6-aa50-2ed6f61bdc98-istio-token\") pod \"e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t\" (UID: \"3ecbed32-91f0-44b6-aa50-2ed6f61bdc98\") " pod="openshift-ingress/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t" Sep 11 18:32:56.302943 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:56.302918 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ecbed32-91f0-44b6-aa50-2ed6f61bdc98-credential-socket\") pod \"e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t\" (UID: \"3ecbed32-91f0-44b6-aa50-2ed6f61bdc98\") " pod="openshift-ingress/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t" Sep 11 18:32:56.303003 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:56.302957 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ecbed32-91f0-44b6-aa50-2ed6f61bdc98-istio-data\") pod \"e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t\" (UID: \"3ecbed32-91f0-44b6-aa50-2ed6f61bdc98\") " pod="openshift-ingress/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t" Sep 11 18:32:56.303003 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:56.302980 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ecbed32-91f0-44b6-aa50-2ed6f61bdc98-workload-socket\") pod \"e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t\" (UID: \"3ecbed32-91f0-44b6-aa50-2ed6f61bdc98\") " pod="openshift-ingress/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t" Sep 11 18:32:56.303267 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:56.303244 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ecbed32-91f0-44b6-aa50-2ed6f61bdc98-istiod-ca-cert\") pod \"e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t\" (UID: \"3ecbed32-91f0-44b6-aa50-2ed6f61bdc98\") " pod="openshift-ingress/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t" Sep 11 18:32:56.304893 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:56.304873 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ecbed32-91f0-44b6-aa50-2ed6f61bdc98-istio-envoy\") pod \"e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t\" (UID: \"3ecbed32-91f0-44b6-aa50-2ed6f61bdc98\") " pod="openshift-ingress/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t" Sep 11 18:32:56.305231 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:56.305210 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ecbed32-91f0-44b6-aa50-2ed6f61bdc98-istio-podinfo\") pod \"e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t\" (UID: \"3ecbed32-91f0-44b6-aa50-2ed6f61bdc98\") " pod="openshift-ingress/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t" Sep 11 18:32:56.310918 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:56.310885 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ecbed32-91f0-44b6-aa50-2ed6f61bdc98-istio-token\") pod \"e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t\" (UID: \"3ecbed32-91f0-44b6-aa50-2ed6f61bdc98\") " pod="openshift-ingress/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t" Sep 11 18:32:56.311038 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:56.310975 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94sd2\" (UniqueName: \"kubernetes.io/projected/3ecbed32-91f0-44b6-aa50-2ed6f61bdc98-kube-api-access-94sd2\") pod \"e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t\" (UID: \"3ecbed32-91f0-44b6-aa50-2ed6f61bdc98\") " pod="openshift-ingress/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t" Sep 11 18:32:56.391200 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:56.391094 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t" Sep 11 18:32:56.693481 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:32:56.693437 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ecbed32_91f0_44b6_aa50_2ed6f61bdc98.slice/crio-c2364f8ec15947ef3fd966c785a6dc29a895461287cfda3b0b4323a6bd3ddd78 WatchSource:0}: Error finding container c2364f8ec15947ef3fd966c785a6dc29a895461287cfda3b0b4323a6bd3ddd78: Status 404 returned error can't find the container with id c2364f8ec15947ef3fd966c785a6dc29a895461287cfda3b0b4323a6bd3ddd78 Sep 11 18:32:56.693481 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:56.693449 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t"] Sep 11 18:32:56.696354 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:56.696312 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:32:56.696452 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:56.696400 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:32:56.696452 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:56.696441 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:32:57.487069 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:57.487005 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t" event={"ID":"3ecbed32-91f0-44b6-aa50-2ed6f61bdc98","Type":"ContainerStarted","Data":"ed8de445da68f2b789e2f88fe1bcf7caf6e5e4be6bf9129dd526e5ce93399c3e"} Sep 11 18:32:57.487554 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:57.487104 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t" event={"ID":"3ecbed32-91f0-44b6-aa50-2ed6f61bdc98","Type":"ContainerStarted","Data":"c2364f8ec15947ef3fd966c785a6dc29a895461287cfda3b0b4323a6bd3ddd78"} Sep 11 18:32:57.509925 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:57.509862 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t" podStartSLOduration=1.509846092 podStartE2EDuration="1.509846092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:32:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:32:57.508641307 +0000 UTC m=+1059.005659069" watchObservedRunningTime="2026-09-11 18:32:57.509846092 +0000 UTC m=+1059.006863846" Sep 11 18:32:58.392113 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:58.392072 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t" Sep 11 18:32:58.397568 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:58.397538 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t" Sep 11 18:32:58.496581 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:58.496550 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t" Sep 11 18:32:58.497544 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:32:58.497521 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t" Sep 11 18:33:04.737585 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:04.737543 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-869176-6c9cfbf7c8-bktwt"] Sep 11 18:33:04.740422 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:04.740396 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-869176-6c9cfbf7c8-bktwt" Sep 11 18:33:04.742614 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:04.742590 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-869176\"" Sep 11 18:33:04.742736 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:04.742700 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-869176-dockercfg-pps6q\"" Sep 11 18:33:04.751763 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:04.751738 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-869176-6c9cfbf7c8-bktwt"] Sep 11 18:33:04.898611 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:04.898557 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzkrp\" (UniqueName: \"kubernetes.io/projected/1b4839f5-19a5-4f78-88d5-99d06b629f30-kube-api-access-kzkrp\") pod \"payload-pre-processing-e2e-shared-a-w2-869176-6c9cfbf7c8-bktwt\" (UID: \"1b4839f5-19a5-4f78-88d5-99d06b629f30\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-869176-6c9cfbf7c8-bktwt" Sep 11 18:33:04.898827 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:04.898714 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b4839f5-19a5-4f78-88d5-99d06b629f30-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-869176-6c9cfbf7c8-bktwt\" (UID: \"1b4839f5-19a5-4f78-88d5-99d06b629f30\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-869176-6c9cfbf7c8-bktwt" Sep 11 18:33:05.000122 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:05.000023 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b4839f5-19a5-4f78-88d5-99d06b629f30-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-869176-6c9cfbf7c8-bktwt\" (UID: \"1b4839f5-19a5-4f78-88d5-99d06b629f30\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-869176-6c9cfbf7c8-bktwt" Sep 11 18:33:05.000280 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:05.000154 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzkrp\" (UniqueName: \"kubernetes.io/projected/1b4839f5-19a5-4f78-88d5-99d06b629f30-kube-api-access-kzkrp\") pod \"payload-pre-processing-e2e-shared-a-w2-869176-6c9cfbf7c8-bktwt\" (UID: \"1b4839f5-19a5-4f78-88d5-99d06b629f30\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-869176-6c9cfbf7c8-bktwt" Sep 11 18:33:05.000928 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:05.000905 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b4839f5-19a5-4f78-88d5-99d06b629f30-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-869176-6c9cfbf7c8-bktwt\" (UID: \"1b4839f5-19a5-4f78-88d5-99d06b629f30\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-869176-6c9cfbf7c8-bktwt" Sep 11 18:33:05.009684 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:05.009656 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzkrp\" (UniqueName: \"kubernetes.io/projected/1b4839f5-19a5-4f78-88d5-99d06b629f30-kube-api-access-kzkrp\") pod \"payload-pre-processing-e2e-shared-a-w2-869176-6c9cfbf7c8-bktwt\" (UID: \"1b4839f5-19a5-4f78-88d5-99d06b629f30\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-869176-6c9cfbf7c8-bktwt" Sep 11 18:33:05.053169 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:05.053131 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-869176-6c9cfbf7c8-bktwt" Sep 11 18:33:05.330229 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:05.330199 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-869176-6c9cfbf7c8-bktwt"] Sep 11 18:33:05.331379 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:33:05.331347 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b4839f5_19a5_4f78_88d5_99d06b629f30.slice/crio-4d2c001c11981da3df615eb64bb75ea4012efccbaa51eb7080eb5f34e3c0de27 WatchSource:0}: Error finding container 4d2c001c11981da3df615eb64bb75ea4012efccbaa51eb7080eb5f34e3c0de27: Status 404 returned error can't find the container with id 4d2c001c11981da3df615eb64bb75ea4012efccbaa51eb7080eb5f34e3c0de27 Sep 11 18:33:05.557333 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:05.557295 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-869176-6c9cfbf7c8-bktwt" event={"ID":"1b4839f5-19a5-4f78-88d5-99d06b629f30","Type":"ContainerStarted","Data":"dc551d9cdc0d44fe40bb52c8b93676cf8bd79e9f1a444cec85500fdd2cbc45f8"} Sep 11 18:33:05.557518 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:05.557353 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-869176-6c9cfbf7c8-bktwt" event={"ID":"1b4839f5-19a5-4f78-88d5-99d06b629f30","Type":"ContainerStarted","Data":"4d2c001c11981da3df615eb64bb75ea4012efccbaa51eb7080eb5f34e3c0de27"} Sep 11 18:33:05.557518 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:05.557379 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-869176-6c9cfbf7c8-bktwt" Sep 11 18:33:05.575143 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:05.575085 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-869176-6c9cfbf7c8-bktwt" podStartSLOduration=1.575050101 podStartE2EDuration="1.575050101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:33:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:33:05.574252526 +0000 UTC m=+1067.071270281" watchObservedRunningTime="2026-09-11 18:33:05.575050101 +0000 UTC m=+1067.072067854" Sep 11 18:33:08.998259 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:08.998218 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c7c83f-6j5zt"] Sep 11 18:33:09.000983 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:09.000965 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c7c83f-6j5zt" Sep 11 18:33:09.003290 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:09.003265 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 18:33:09.003427 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:09.003271 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n968b\"" Sep 11 18:33:09.009622 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:09.009590 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c7c83f-6j5zt"] Sep 11 18:33:09.143257 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:09.143226 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfac7899-3201-4e52-9382-978d1eb07d30-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-c7c83f-6j5zt\" (UID: \"cfac7899-3201-4e52-9382-978d1eb07d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c7c83f-6j5zt" Sep 11 18:33:09.244442 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:09.244398 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfac7899-3201-4e52-9382-978d1eb07d30-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-c7c83f-6j5zt\" (UID: \"cfac7899-3201-4e52-9382-978d1eb07d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c7c83f-6j5zt" Sep 11 18:33:09.245206 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:09.245186 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfac7899-3201-4e52-9382-978d1eb07d30-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-c7c83f-6j5zt\" (UID: \"cfac7899-3201-4e52-9382-978d1eb07d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c7c83f-6j5zt" Sep 11 18:33:09.313363 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:09.313283 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c7c83f-6j5zt" Sep 11 18:33:09.588752 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:09.588719 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c7c83f-6j5zt"] Sep 11 18:33:09.589169 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:33:09.589142 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfac7899_3201_4e52_9382_978d1eb07d30.slice/crio-06352fcd9679544174922c2bb305e2d3d48aca56ec7fb52f2708fd54f0ccaedb WatchSource:0}: Error finding container 06352fcd9679544174922c2bb305e2d3d48aca56ec7fb52f2708fd54f0ccaedb: Status 404 returned error can't find the container with id 06352fcd9679544174922c2bb305e2d3d48aca56ec7fb52f2708fd54f0ccaedb Sep 11 18:33:10.596087 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:10.595973 2561 generic.go:356] "Generic (PLEG): container finished" podID="cfac7899-3201-4e52-9382-978d1eb07d30" containerID="e4af4faac8d565388591073cb505cfbde84362d61595a2b17c10ebc7e04913c7" exitCode=0 Sep 11 18:33:10.596087 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:10.596028 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c7c83f-6j5zt" event={"ID":"cfac7899-3201-4e52-9382-978d1eb07d30","Type":"ContainerDied","Data":"e4af4faac8d565388591073cb505cfbde84362d61595a2b17c10ebc7e04913c7"} Sep 11 18:33:10.596087 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:10.596084 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c7c83f-6j5zt" event={"ID":"cfac7899-3201-4e52-9382-978d1eb07d30","Type":"ContainerStarted","Data":"06352fcd9679544174922c2bb305e2d3d48aca56ec7fb52f2708fd54f0ccaedb"} Sep 11 18:33:11.756424 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:11.756392 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c7c83f-6j5zt" Sep 11 18:33:11.872913 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:11.872819 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfac7899-3201-4e52-9382-978d1eb07d30-maas-api-service-ca\") pod \"cfac7899-3201-4e52-9382-978d1eb07d30\" (UID: \"cfac7899-3201-4e52-9382-978d1eb07d30\") " Sep 11 18:33:11.873225 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:11.873200 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfac7899-3201-4e52-9382-978d1eb07d30-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cfac7899-3201-4e52-9382-978d1eb07d30" (UID: "cfac7899-3201-4e52-9382-978d1eb07d30"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:33:11.974324 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:11.974286 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfac7899-3201-4e52-9382-978d1eb07d30-maas-api-service-ca\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:33:12.618938 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:12.618902 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c7c83f-6j5zt" event={"ID":"cfac7899-3201-4e52-9382-978d1eb07d30","Type":"ContainerDied","Data":"06352fcd9679544174922c2bb305e2d3d48aca56ec7fb52f2708fd54f0ccaedb"} Sep 11 18:33:12.619170 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:12.618941 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c7c83f-6j5zt" Sep 11 18:33:12.619170 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:12.618951 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06352fcd9679544174922c2bb305e2d3d48aca56ec7fb52f2708fd54f0ccaedb" Sep 11 18:33:16.567236 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:16.567206 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-869176-6c9cfbf7c8-bktwt" Sep 11 18:33:20.029557 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:20.029516 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc"] Sep 11 18:33:20.030115 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:20.029999 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfac7899-3201-4e52-9382-978d1eb07d30" containerName="revoke-keys" Sep 11 18:33:20.030115 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:20.030013 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfac7899-3201-4e52-9382-978d1eb07d30" containerName="revoke-keys" Sep 11 18:33:20.030239 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:20.030144 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfac7899-3201-4e52-9382-978d1eb07d30" containerName="revoke-keys" Sep 11 18:33:20.041198 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:20.041167 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc" Sep 11 18:33:20.043779 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:20.043583 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-0ead89-openshift-default-dockercfg-dptl7\"" Sep 11 18:33:20.045078 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:20.044947 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc"] Sep 11 18:33:20.060659 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:20.060629 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b79f3a4c-779f-4157-9746-bfe9348292b3-istio-podinfo\") pod \"e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc\" (UID: \"b79f3a4c-779f-4157-9746-bfe9348292b3\") " pod="openshift-ingress/e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc" Sep 11 18:33:20.060833 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:20.060683 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b79f3a4c-779f-4157-9746-bfe9348292b3-credential-socket\") pod \"e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc\" (UID: \"b79f3a4c-779f-4157-9746-bfe9348292b3\") " pod="openshift-ingress/e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc" Sep 11 18:33:20.060833 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:20.060740 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b79f3a4c-779f-4157-9746-bfe9348292b3-workload-certs\") pod \"e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc\" (UID: \"b79f3a4c-779f-4157-9746-bfe9348292b3\") " pod="openshift-ingress/e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc" Sep 11 18:33:20.060833 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:20.060775 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4sjs\" (UniqueName: \"kubernetes.io/projected/b79f3a4c-779f-4157-9746-bfe9348292b3-kube-api-access-d4sjs\") pod \"e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc\" (UID: \"b79f3a4c-779f-4157-9746-bfe9348292b3\") " pod="openshift-ingress/e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc" Sep 11 18:33:20.060833 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:20.060816 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b79f3a4c-779f-4157-9746-bfe9348292b3-istio-token\") pod \"e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc\" (UID: \"b79f3a4c-779f-4157-9746-bfe9348292b3\") " pod="openshift-ingress/e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc" Sep 11 18:33:20.061074 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:20.060870 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b79f3a4c-779f-4157-9746-bfe9348292b3-istio-data\") pod \"e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc\" (UID: \"b79f3a4c-779f-4157-9746-bfe9348292b3\") " pod="openshift-ingress/e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc" Sep 11 18:33:20.061074 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:20.060899 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b79f3a4c-779f-4157-9746-bfe9348292b3-istio-envoy\") pod \"e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc\" (UID: \"b79f3a4c-779f-4157-9746-bfe9348292b3\") " pod="openshift-ingress/e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc" Sep 11 18:33:20.061074 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:20.060932 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b79f3a4c-779f-4157-9746-bfe9348292b3-workload-socket\") pod \"e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc\" (UID: \"b79f3a4c-779f-4157-9746-bfe9348292b3\") " pod="openshift-ingress/e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc" Sep 11 18:33:20.061074 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:20.060986 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b79f3a4c-779f-4157-9746-bfe9348292b3-istiod-ca-cert\") pod \"e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc\" (UID: \"b79f3a4c-779f-4157-9746-bfe9348292b3\") " pod="openshift-ingress/e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc" Sep 11 18:33:20.162343 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:20.162303 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b79f3a4c-779f-4157-9746-bfe9348292b3-workload-certs\") pod \"e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc\" (UID: \"b79f3a4c-779f-4157-9746-bfe9348292b3\") " pod="openshift-ingress/e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc" Sep 11 18:33:20.162640 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:20.162616 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4sjs\" (UniqueName: \"kubernetes.io/projected/b79f3a4c-779f-4157-9746-bfe9348292b3-kube-api-access-d4sjs\") pod \"e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc\" (UID: \"b79f3a4c-779f-4157-9746-bfe9348292b3\") " pod="openshift-ingress/e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc" Sep 11 18:33:20.162777 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:20.162758 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b79f3a4c-779f-4157-9746-bfe9348292b3-workload-certs\") pod \"e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc\" (UID: \"b79f3a4c-779f-4157-9746-bfe9348292b3\") " pod="openshift-ingress/e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc" Sep 11 18:33:20.162888 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:20.162868 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b79f3a4c-779f-4157-9746-bfe9348292b3-istio-token\") pod \"e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc\" (UID: \"b79f3a4c-779f-4157-9746-bfe9348292b3\") " pod="openshift-ingress/e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc" Sep 11 18:33:20.163135 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:20.163103 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b79f3a4c-779f-4157-9746-bfe9348292b3-istio-data\") pod \"e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc\" (UID: \"b79f3a4c-779f-4157-9746-bfe9348292b3\") " pod="openshift-ingress/e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc" Sep 11 18:33:20.163294 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:20.163277 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b79f3a4c-779f-4157-9746-bfe9348292b3-istio-envoy\") pod \"e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc\" (UID: \"b79f3a4c-779f-4157-9746-bfe9348292b3\") " pod="openshift-ingress/e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc" Sep 11 18:33:20.163420 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:20.163404 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b79f3a4c-779f-4157-9746-bfe9348292b3-workload-socket\") pod \"e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc\" (UID: \"b79f3a4c-779f-4157-9746-bfe9348292b3\") " pod="openshift-ingress/e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc" Sep 11 18:33:20.163703 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:20.163681 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b79f3a4c-779f-4157-9746-bfe9348292b3-istio-data\") pod \"e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc\" (UID: \"b79f3a4c-779f-4157-9746-bfe9348292b3\") " pod="openshift-ingress/e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc" Sep 11 18:33:20.163862 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:20.163831 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b79f3a4c-779f-4157-9746-bfe9348292b3-workload-socket\") pod \"e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc\" (UID: \"b79f3a4c-779f-4157-9746-bfe9348292b3\") " pod="openshift-ingress/e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc" Sep 11 18:33:20.163961 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:20.163871 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b79f3a4c-779f-4157-9746-bfe9348292b3-istiod-ca-cert\") pod \"e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc\" (UID: \"b79f3a4c-779f-4157-9746-bfe9348292b3\") " pod="openshift-ingress/e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc" Sep 11 18:33:20.164025 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:20.163979 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b79f3a4c-779f-4157-9746-bfe9348292b3-istio-podinfo\") pod \"e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc\" (UID: \"b79f3a4c-779f-4157-9746-bfe9348292b3\") " pod="openshift-ingress/e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc" Sep 11 18:33:20.164106 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:20.164029 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b79f3a4c-779f-4157-9746-bfe9348292b3-credential-socket\") pod \"e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc\" (UID: \"b79f3a4c-779f-4157-9746-bfe9348292b3\") " pod="openshift-ingress/e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc" Sep 11 18:33:20.164386 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:20.164355 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b79f3a4c-779f-4157-9746-bfe9348292b3-credential-socket\") pod \"e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc\" (UID: \"b79f3a4c-779f-4157-9746-bfe9348292b3\") " pod="openshift-ingress/e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc" Sep 11 18:33:20.164611 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:20.164591 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b79f3a4c-779f-4157-9746-bfe9348292b3-istiod-ca-cert\") pod \"e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc\" (UID: \"b79f3a4c-779f-4157-9746-bfe9348292b3\") " pod="openshift-ingress/e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc" Sep 11 18:33:20.166280 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:20.166191 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b79f3a4c-779f-4157-9746-bfe9348292b3-istio-envoy\") pod \"e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc\" (UID: \"b79f3a4c-779f-4157-9746-bfe9348292b3\") " pod="openshift-ingress/e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc" Sep 11 18:33:20.166756 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:20.166720 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b79f3a4c-779f-4157-9746-bfe9348292b3-istio-podinfo\") pod \"e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc\" (UID: \"b79f3a4c-779f-4157-9746-bfe9348292b3\") " pod="openshift-ingress/e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc" Sep 11 18:33:20.171282 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:20.171097 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b79f3a4c-779f-4157-9746-bfe9348292b3-istio-token\") pod \"e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc\" (UID: \"b79f3a4c-779f-4157-9746-bfe9348292b3\") " pod="openshift-ingress/e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc" Sep 11 18:33:20.171282 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:20.171250 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4sjs\" (UniqueName: \"kubernetes.io/projected/b79f3a4c-779f-4157-9746-bfe9348292b3-kube-api-access-d4sjs\") pod \"e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc\" (UID: \"b79f3a4c-779f-4157-9746-bfe9348292b3\") " pod="openshift-ingress/e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc" Sep 11 18:33:20.355146 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:20.355030 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc" Sep 11 18:33:20.670987 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:33:20.670941 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb79f3a4c_779f_4157_9746_bfe9348292b3.slice/crio-5dbbff5d7bc809dd9780b20521863e58cdc46a7f98810c0435e2c633f53068a0 WatchSource:0}: Error finding container 5dbbff5d7bc809dd9780b20521863e58cdc46a7f98810c0435e2c633f53068a0: Status 404 returned error can't find the container with id 5dbbff5d7bc809dd9780b20521863e58cdc46a7f98810c0435e2c633f53068a0 Sep 11 18:33:20.672995 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:20.672963 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc"] Sep 11 18:33:20.674204 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:20.674169 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:33:20.674336 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:20.674231 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:33:20.674336 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:20.674261 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:33:21.553910 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:21.553868 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c7c83f-6dcc6d86cc-mpw6w"] Sep 11 18:33:21.554420 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:21.554162 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-c7c83f-6dcc6d86cc-mpw6w" podUID="76bf21cf-8897-4f9c-a22c-c7f437ff4d19" containerName="payload-processing" containerID="cri-o://65730be3b932651859cfff6a1c64870ce6acff2e18a3e9cbea6de5a5510272e6" gracePeriod=30 Sep 11 18:33:21.689174 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:21.688959 2561 generic.go:356] "Generic (PLEG): container finished" podID="76bf21cf-8897-4f9c-a22c-c7f437ff4d19" containerID="65730be3b932651859cfff6a1c64870ce6acff2e18a3e9cbea6de5a5510272e6" exitCode=0 Sep 11 18:33:21.689174 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:21.689094 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-c7c83f-6dcc6d86cc-mpw6w" event={"ID":"76bf21cf-8897-4f9c-a22c-c7f437ff4d19","Type":"ContainerDied","Data":"65730be3b932651859cfff6a1c64870ce6acff2e18a3e9cbea6de5a5510272e6"} Sep 11 18:33:21.692210 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:21.691885 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc" event={"ID":"b79f3a4c-779f-4157-9746-bfe9348292b3","Type":"ContainerStarted","Data":"64677ebfe10d115e7faddd914b169ff5d6921b41ba0b7b4c31cc8ee39fe1f91b"} Sep 11 18:33:21.692210 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:21.691943 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc" event={"ID":"b79f3a4c-779f-4157-9746-bfe9348292b3","Type":"ContainerStarted","Data":"5dbbff5d7bc809dd9780b20521863e58cdc46a7f98810c0435e2c633f53068a0"} Sep 11 18:33:21.711457 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:21.711335 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc" podStartSLOduration=1.7113109359999998 podStartE2EDuration="1.711310936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:33:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:33:21.710917887 +0000 UTC m=+1083.207935632" watchObservedRunningTime="2026-09-11 18:33:21.711310936 +0000 UTC m=+1083.208328692" Sep 11 18:33:21.746187 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:21.746113 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c7c83f-6j5zt"] Sep 11 18:33:21.751135 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:21.751097 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c7c83f-6j5zt"] Sep 11 18:33:21.838554 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:21.838525 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-c7c83f-6dcc6d86cc-mpw6w" Sep 11 18:33:21.908131 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:21.901487 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76bf21cf-8897-4f9c-a22c-c7f437ff4d19-plugins-config-volume\") pod \"76bf21cf-8897-4f9c-a22c-c7f437ff4d19\" (UID: \"76bf21cf-8897-4f9c-a22c-c7f437ff4d19\") " Sep 11 18:33:21.908131 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:21.901551 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2skmq\" (UniqueName: \"kubernetes.io/projected/76bf21cf-8897-4f9c-a22c-c7f437ff4d19-kube-api-access-2skmq\") pod \"76bf21cf-8897-4f9c-a22c-c7f437ff4d19\" (UID: \"76bf21cf-8897-4f9c-a22c-c7f437ff4d19\") " Sep 11 18:33:21.908131 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:21.902477 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76bf21cf-8897-4f9c-a22c-c7f437ff4d19-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "76bf21cf-8897-4f9c-a22c-c7f437ff4d19" (UID: "76bf21cf-8897-4f9c-a22c-c7f437ff4d19"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:33:21.922786 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:21.922730 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76bf21cf-8897-4f9c-a22c-c7f437ff4d19-kube-api-access-2skmq" (OuterVolumeSpecName: "kube-api-access-2skmq") pod "76bf21cf-8897-4f9c-a22c-c7f437ff4d19" (UID: "76bf21cf-8897-4f9c-a22c-c7f437ff4d19"). InnerVolumeSpecName "kube-api-access-2skmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:33:22.003378 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:22.003333 2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76bf21cf-8897-4f9c-a22c-c7f437ff4d19-plugins-config-volume\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:33:22.003378 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:22.003373 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2skmq\" (UniqueName: \"kubernetes.io/projected/76bf21cf-8897-4f9c-a22c-c7f437ff4d19-kube-api-access-2skmq\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:33:22.355990 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:22.355952 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc" Sep 11 18:33:22.362476 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:22.362440 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc" Sep 11 18:33:22.701297 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:22.701266 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-c7c83f-6dcc6d86cc-mpw6w" Sep 11 18:33:22.701297 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:22.701293 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-c7c83f-6dcc6d86cc-mpw6w" event={"ID":"76bf21cf-8897-4f9c-a22c-c7f437ff4d19","Type":"ContainerDied","Data":"e73dd404572aec125826144f38d1406569343eaf4b0f79dc960d1f3847798485"} Sep 11 18:33:22.701863 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:22.701356 2561 scope.go:117] "RemoveContainer" containerID="65730be3b932651859cfff6a1c64870ce6acff2e18a3e9cbea6de5a5510272e6" Sep 11 18:33:22.701931 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:22.701866 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc" Sep 11 18:33:22.702915 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:22.702897 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc" Sep 11 18:33:22.749616 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:22.748782 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c7c83f-6dcc6d86cc-mpw6w"] Sep 11 18:33:22.754574 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:22.754545 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c7c83f-6dcc6d86cc-mpw6w"] Sep 11 18:33:23.141697 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:23.141614 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76bf21cf-8897-4f9c-a22c-c7f437ff4d19" path="/var/lib/kubelet/pods/76bf21cf-8897-4f9c-a22c-c7f437ff4d19/volumes" Sep 11 18:33:23.141991 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:23.141977 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfac7899-3201-4e52-9382-978d1eb07d30" path="/var/lib/kubelet/pods/cfac7899-3201-4e52-9382-978d1eb07d30/volumes" Sep 11 18:33:24.728803 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:24.728769 2561 generic.go:356] "Generic (PLEG): container finished" podID="3ecbed32-91f0-44b6-aa50-2ed6f61bdc98" containerID="ed8de445da68f2b789e2f88fe1bcf7caf6e5e4be6bf9129dd526e5ce93399c3e" exitCode=0 Sep 11 18:33:24.729339 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:24.728859 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t" event={"ID":"3ecbed32-91f0-44b6-aa50-2ed6f61bdc98","Type":"ContainerDied","Data":"ed8de445da68f2b789e2f88fe1bcf7caf6e5e4be6bf9129dd526e5ce93399c3e"} Sep 11 18:33:24.729339 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:24.729301 2561 scope.go:117] "RemoveContainer" containerID="ed8de445da68f2b789e2f88fe1bcf7caf6e5e4be6bf9129dd526e5ce93399c3e" Sep 11 18:33:24.730396 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:24.730369 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:33:24.730474 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:24.730435 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:33:24.730474 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:24.730469 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:33:25.738877 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:25.738844 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t" event={"ID":"3ecbed32-91f0-44b6-aa50-2ed6f61bdc98","Type":"ContainerStarted","Data":"bc05afa511da53b8afe052665faabcd120ca9c792dff1261494428dab65c9623"} Sep 11 18:33:26.392193 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:26.392157 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t" Sep 11 18:33:26.393583 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:26.393550 2561 patch_prober.go:28] interesting pod/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Sep 11 18:33:26.393792 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:26.393765 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t" podUID="3ecbed32-91f0-44b6-aa50-2ed6f61bdc98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Sep 11 18:33:27.392592 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:27.392552 2561 patch_prober.go:28] interesting pod/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Sep 11 18:33:27.393627 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:27.393157 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t" podUID="3ecbed32-91f0-44b6-aa50-2ed6f61bdc98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Sep 11 18:33:28.174578 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:28.174543 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-0ead89-65d6c74f65-2tx94"] Sep 11 18:33:28.175248 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:28.175228 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76bf21cf-8897-4f9c-a22c-c7f437ff4d19" containerName="payload-processing" Sep 11 18:33:28.175360 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:28.175350 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="76bf21cf-8897-4f9c-a22c-c7f437ff4d19" containerName="payload-processing" Sep 11 18:33:28.175518 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:28.175509 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="76bf21cf-8897-4f9c-a22c-c7f437ff4d19" containerName="payload-processing" Sep 11 18:33:28.177795 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:28.177778 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0ead89-65d6c74f65-2tx94" Sep 11 18:33:28.180196 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:28.180174 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-0ead89\"" Sep 11 18:33:28.180474 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:28.180182 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-0ead89-dockercfg-t89xd\"" Sep 11 18:33:28.198082 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:28.197566 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-0ead89-65d6c74f65-2tx94"] Sep 11 18:33:28.271112 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:28.270678 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lff4j\" (UniqueName: \"kubernetes.io/projected/e2c31ea5-d698-47c7-b149-ee08f7eb184e-kube-api-access-lff4j\") pod \"payload-processing-e2e-shared-b-w2-0ead89-65d6c74f65-2tx94\" (UID: \"e2c31ea5-d698-47c7-b149-ee08f7eb184e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0ead89-65d6c74f65-2tx94" Sep 11 18:33:28.271112 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:28.270815 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2c31ea5-d698-47c7-b149-ee08f7eb184e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-0ead89-65d6c74f65-2tx94\" (UID: \"e2c31ea5-d698-47c7-b149-ee08f7eb184e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0ead89-65d6c74f65-2tx94" Sep 11 18:33:28.372397 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:28.372357 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lff4j\" (UniqueName: \"kubernetes.io/projected/e2c31ea5-d698-47c7-b149-ee08f7eb184e-kube-api-access-lff4j\") pod \"payload-processing-e2e-shared-b-w2-0ead89-65d6c74f65-2tx94\" (UID: \"e2c31ea5-d698-47c7-b149-ee08f7eb184e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0ead89-65d6c74f65-2tx94" Sep 11 18:33:28.372546 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:28.372447 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2c31ea5-d698-47c7-b149-ee08f7eb184e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-0ead89-65d6c74f65-2tx94\" (UID: \"e2c31ea5-d698-47c7-b149-ee08f7eb184e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0ead89-65d6c74f65-2tx94" Sep 11 18:33:28.373112 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:28.373092 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2c31ea5-d698-47c7-b149-ee08f7eb184e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-0ead89-65d6c74f65-2tx94\" (UID: \"e2c31ea5-d698-47c7-b149-ee08f7eb184e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0ead89-65d6c74f65-2tx94" Sep 11 18:33:28.381075 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:28.381035 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lff4j\" (UniqueName: \"kubernetes.io/projected/e2c31ea5-d698-47c7-b149-ee08f7eb184e-kube-api-access-lff4j\") pod \"payload-processing-e2e-shared-b-w2-0ead89-65d6c74f65-2tx94\" (UID: \"e2c31ea5-d698-47c7-b149-ee08f7eb184e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0ead89-65d6c74f65-2tx94" Sep 11 18:33:28.392167 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:28.392125 2561 patch_prober.go:28] interesting pod/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Sep 11 18:33:28.392365 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:28.392208 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t" podUID="3ecbed32-91f0-44b6-aa50-2ed6f61bdc98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Sep 11 18:33:28.497633 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:28.497533 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t" Sep 11 18:33:28.509298 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:28.509258 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0ead89-65d6c74f65-2tx94" Sep 11 18:33:28.794811 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:28.794752 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-0ead89-65d6c74f65-2tx94"] Sep 11 18:33:28.798371 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:33:28.798320 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2c31ea5_d698_47c7_b149_ee08f7eb184e.slice/crio-aaeb90ca0c952a2c6b6236e890b4881595107dec4c19325b475551b7a6842432 WatchSource:0}: Error finding container aaeb90ca0c952a2c6b6236e890b4881595107dec4c19325b475551b7a6842432: Status 404 returned error can't find the container with id aaeb90ca0c952a2c6b6236e890b4881595107dec4c19325b475551b7a6842432 Sep 11 18:33:29.392290 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:29.392246 2561 patch_prober.go:28] interesting pod/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Sep 11 18:33:29.392488 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:29.392316 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t" podUID="3ecbed32-91f0-44b6-aa50-2ed6f61bdc98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Sep 11 18:33:29.767457 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:29.767332 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0ead89-65d6c74f65-2tx94" event={"ID":"e2c31ea5-d698-47c7-b149-ee08f7eb184e","Type":"ContainerStarted","Data":"49d69abb877ce8172e13d1a6141e31196269b34460090e142605a9d433db9161"} Sep 11 18:33:29.767457 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:29.767410 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0ead89-65d6c74f65-2tx94" event={"ID":"e2c31ea5-d698-47c7-b149-ee08f7eb184e","Type":"ContainerStarted","Data":"aaeb90ca0c952a2c6b6236e890b4881595107dec4c19325b475551b7a6842432"} Sep 11 18:33:29.767457 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:29.767432 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0ead89-65d6c74f65-2tx94" Sep 11 18:33:29.790329 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:29.789302 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0ead89-65d6c74f65-2tx94" podStartSLOduration=1.7892786470000002 podStartE2EDuration="1.789278647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:33:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:33:29.783790791 +0000 UTC m=+1091.280808545" watchObservedRunningTime="2026-09-11 18:33:29.789278647 +0000 UTC m=+1091.286296404" Sep 11 18:33:29.868743 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:29.868710 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm"] Sep 11 18:33:29.869125 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:29.868917 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm" podUID="1ab46e7b-3161-46a7-9603-629d75d31c1d" containerName="istio-proxy" containerID="cri-o://dec4d2d366455dbae3f4090c9fa6df16dc504414765e7752ae8c704231c036a7" gracePeriod=30 Sep 11 18:33:30.392490 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:30.392445 2561 patch_prober.go:28] interesting pod/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Sep 11 18:33:30.392706 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:30.392532 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t" podUID="3ecbed32-91f0-44b6-aa50-2ed6f61bdc98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Sep 11 18:33:31.395847 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:31.395812 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t" Sep 11 18:33:31.397083 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:31.397047 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t" Sep 11 18:33:35.131684 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:35.131621 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm" Sep 11 18:33:35.268225 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:35.268166 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ab46e7b-3161-46a7-9603-629d75d31c1d-istio-envoy\") pod \"1ab46e7b-3161-46a7-9603-629d75d31c1d\" (UID: \"1ab46e7b-3161-46a7-9603-629d75d31c1d\") " Sep 11 18:33:35.268420 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:35.268241 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ab46e7b-3161-46a7-9603-629d75d31c1d-istio-token\") pod \"1ab46e7b-3161-46a7-9603-629d75d31c1d\" (UID: \"1ab46e7b-3161-46a7-9603-629d75d31c1d\") " Sep 11 18:33:35.268420 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:35.268289 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ab46e7b-3161-46a7-9603-629d75d31c1d-workload-socket\") pod \"1ab46e7b-3161-46a7-9603-629d75d31c1d\" (UID: \"1ab46e7b-3161-46a7-9603-629d75d31c1d\") " Sep 11 18:33:35.268420 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:35.268336 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vdgt8\" (UniqueName: \"kubernetes.io/projected/1ab46e7b-3161-46a7-9603-629d75d31c1d-kube-api-access-vdgt8\") pod \"1ab46e7b-3161-46a7-9603-629d75d31c1d\" (UID: \"1ab46e7b-3161-46a7-9603-629d75d31c1d\") " Sep 11 18:33:35.268420 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:35.268367 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ab46e7b-3161-46a7-9603-629d75d31c1d-istio-data\") pod \"1ab46e7b-3161-46a7-9603-629d75d31c1d\" (UID: \"1ab46e7b-3161-46a7-9603-629d75d31c1d\") " Sep 11 18:33:35.268649 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:35.268443 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ab46e7b-3161-46a7-9603-629d75d31c1d-istiod-ca-cert\") pod \"1ab46e7b-3161-46a7-9603-629d75d31c1d\" (UID: \"1ab46e7b-3161-46a7-9603-629d75d31c1d\") " Sep 11 18:33:35.268649 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:35.268503 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ab46e7b-3161-46a7-9603-629d75d31c1d-istio-podinfo\") pod \"1ab46e7b-3161-46a7-9603-629d75d31c1d\" (UID: \"1ab46e7b-3161-46a7-9603-629d75d31c1d\") " Sep 11 18:33:35.268649 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:35.268536 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ab46e7b-3161-46a7-9603-629d75d31c1d-workload-certs\") pod \"1ab46e7b-3161-46a7-9603-629d75d31c1d\" (UID: \"1ab46e7b-3161-46a7-9603-629d75d31c1d\") " Sep 11 18:33:35.268649 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:35.268592 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ab46e7b-3161-46a7-9603-629d75d31c1d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1ab46e7b-3161-46a7-9603-629d75d31c1d" (UID: "1ab46e7b-3161-46a7-9603-629d75d31c1d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:33:35.268649 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:35.268630 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ab46e7b-3161-46a7-9603-629d75d31c1d-credential-socket\") pod \"1ab46e7b-3161-46a7-9603-629d75d31c1d\" (UID: \"1ab46e7b-3161-46a7-9603-629d75d31c1d\") " Sep 11 18:33:35.268915 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:35.268705 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ab46e7b-3161-46a7-9603-629d75d31c1d-istio-data" (OuterVolumeSpecName: "istio-data") pod "1ab46e7b-3161-46a7-9603-629d75d31c1d" (UID: "1ab46e7b-3161-46a7-9603-629d75d31c1d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:33:35.268915 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:35.268788 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ab46e7b-3161-46a7-9603-629d75d31c1d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1ab46e7b-3161-46a7-9603-629d75d31c1d" (UID: "1ab46e7b-3161-46a7-9603-629d75d31c1d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:33:35.269031 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:35.268924 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ab46e7b-3161-46a7-9603-629d75d31c1d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1ab46e7b-3161-46a7-9603-629d75d31c1d" (UID: "1ab46e7b-3161-46a7-9603-629d75d31c1d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:33:35.269031 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:35.269007 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ab46e7b-3161-46a7-9603-629d75d31c1d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1ab46e7b-3161-46a7-9603-629d75d31c1d" (UID: "1ab46e7b-3161-46a7-9603-629d75d31c1d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:33:35.269242 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:35.269210 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ab46e7b-3161-46a7-9603-629d75d31c1d-istiod-ca-cert\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:33:35.269242 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:35.269241 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ab46e7b-3161-46a7-9603-629d75d31c1d-workload-certs\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:33:35.269364 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:35.269257 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ab46e7b-3161-46a7-9603-629d75d31c1d-credential-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:33:35.269364 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:35.269273 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ab46e7b-3161-46a7-9603-629d75d31c1d-workload-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:33:35.269364 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:35.269290 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ab46e7b-3161-46a7-9603-629d75d31c1d-istio-data\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:33:35.270603 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:35.270576 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ab46e7b-3161-46a7-9603-629d75d31c1d-istio-token" (OuterVolumeSpecName: "istio-token") pod "1ab46e7b-3161-46a7-9603-629d75d31c1d" (UID: "1ab46e7b-3161-46a7-9603-629d75d31c1d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:33:35.270727 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:35.270611 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ab46e7b-3161-46a7-9603-629d75d31c1d-kube-api-access-vdgt8" (OuterVolumeSpecName: "kube-api-access-vdgt8") pod "1ab46e7b-3161-46a7-9603-629d75d31c1d" (UID: "1ab46e7b-3161-46a7-9603-629d75d31c1d"). InnerVolumeSpecName "kube-api-access-vdgt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:33:35.270912 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:35.270884 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ab46e7b-3161-46a7-9603-629d75d31c1d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1ab46e7b-3161-46a7-9603-629d75d31c1d" (UID: "1ab46e7b-3161-46a7-9603-629d75d31c1d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:33:35.270912 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:35.270908 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ab46e7b-3161-46a7-9603-629d75d31c1d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1ab46e7b-3161-46a7-9603-629d75d31c1d" (UID: "1ab46e7b-3161-46a7-9603-629d75d31c1d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 18:33:35.370866 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:35.370814 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ab46e7b-3161-46a7-9603-629d75d31c1d-istio-podinfo\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:33:35.370866 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:35.370857 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ab46e7b-3161-46a7-9603-629d75d31c1d-istio-envoy\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:33:35.370866 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:35.370867 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ab46e7b-3161-46a7-9603-629d75d31c1d-istio-token\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:33:35.370866 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:35.370876 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdgt8\" (UniqueName: \"kubernetes.io/projected/1ab46e7b-3161-46a7-9603-629d75d31c1d-kube-api-access-vdgt8\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:33:35.815882 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:35.815844 2561 generic.go:356] "Generic (PLEG): container finished" podID="1ab46e7b-3161-46a7-9603-629d75d31c1d" containerID="dec4d2d366455dbae3f4090c9fa6df16dc504414765e7752ae8c704231c036a7" exitCode=0 Sep 11 18:33:35.816081 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:35.815892 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm" event={"ID":"1ab46e7b-3161-46a7-9603-629d75d31c1d","Type":"ContainerDied","Data":"dec4d2d366455dbae3f4090c9fa6df16dc504414765e7752ae8c704231c036a7"} Sep 11 18:33:35.816081 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:35.815902 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm" Sep 11 18:33:35.816081 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:35.815935 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm" event={"ID":"1ab46e7b-3161-46a7-9603-629d75d31c1d","Type":"ContainerDied","Data":"11a82d7813c712b6b9fb8876147ef51790d11729565d53111fb7709f46926296"} Sep 11 18:33:35.816081 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:35.815956 2561 scope.go:117] "RemoveContainer" containerID="dec4d2d366455dbae3f4090c9fa6df16dc504414765e7752ae8c704231c036a7" Sep 11 18:33:35.829208 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:35.829185 2561 scope.go:117] "RemoveContainer" containerID="d108db93c1153fe7bcaa3cacfa3d4c4298feeacdf22a149ef107297d56e356e6" Sep 11 18:33:35.838756 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:35.838737 2561 scope.go:117] "RemoveContainer" containerID="dec4d2d366455dbae3f4090c9fa6df16dc504414765e7752ae8c704231c036a7" Sep 11 18:33:35.839019 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:33:35.839001 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dec4d2d366455dbae3f4090c9fa6df16dc504414765e7752ae8c704231c036a7\": container with ID starting with dec4d2d366455dbae3f4090c9fa6df16dc504414765e7752ae8c704231c036a7 not found: ID does not exist" containerID="dec4d2d366455dbae3f4090c9fa6df16dc504414765e7752ae8c704231c036a7" Sep 11 18:33:35.839079 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:35.839028 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dec4d2d366455dbae3f4090c9fa6df16dc504414765e7752ae8c704231c036a7"} err="failed to get container status \"dec4d2d366455dbae3f4090c9fa6df16dc504414765e7752ae8c704231c036a7\": rpc error: code = NotFound desc = could not find container \"dec4d2d366455dbae3f4090c9fa6df16dc504414765e7752ae8c704231c036a7\": container with ID starting with dec4d2d366455dbae3f4090c9fa6df16dc504414765e7752ae8c704231c036a7 not found: ID does not exist" Sep 11 18:33:35.839079 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:35.839046 2561 scope.go:117] "RemoveContainer" containerID="d108db93c1153fe7bcaa3cacfa3d4c4298feeacdf22a149ef107297d56e356e6" Sep 11 18:33:35.839341 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:33:35.839326 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d108db93c1153fe7bcaa3cacfa3d4c4298feeacdf22a149ef107297d56e356e6\": container with ID starting with d108db93c1153fe7bcaa3cacfa3d4c4298feeacdf22a149ef107297d56e356e6 not found: ID does not exist" containerID="d108db93c1153fe7bcaa3cacfa3d4c4298feeacdf22a149ef107297d56e356e6" Sep 11 18:33:35.839383 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:35.839345 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d108db93c1153fe7bcaa3cacfa3d4c4298feeacdf22a149ef107297d56e356e6"} err="failed to get container status \"d108db93c1153fe7bcaa3cacfa3d4c4298feeacdf22a149ef107297d56e356e6\": rpc error: code = NotFound desc = could not find container \"d108db93c1153fe7bcaa3cacfa3d4c4298feeacdf22a149ef107297d56e356e6\": container with ID starting with d108db93c1153fe7bcaa3cacfa3d4c4298feeacdf22a149ef107297d56e356e6 not found: ID does not exist" Sep 11 18:33:35.847655 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:35.847628 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm"] Sep 11 18:33:35.850537 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:35.850517 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-c7c83f-openshift-default-b765f76bd-8tvdm"] Sep 11 18:33:37.139945 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:37.139909 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ab46e7b-3161-46a7-9603-629d75d31c1d" path="/var/lib/kubelet/pods/1ab46e7b-3161-46a7-9603-629d75d31c1d/volumes" Sep 11 18:33:40.777669 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:40.777634 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0ead89-65d6c74f65-2tx94" Sep 11 18:33:49.940930 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:49.940895 2561 generic.go:356] "Generic (PLEG): container finished" podID="b79f3a4c-779f-4157-9746-bfe9348292b3" containerID="64677ebfe10d115e7faddd914b169ff5d6921b41ba0b7b4c31cc8ee39fe1f91b" exitCode=0 Sep 11 18:33:49.941373 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:49.940975 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc" event={"ID":"b79f3a4c-779f-4157-9746-bfe9348292b3","Type":"ContainerDied","Data":"64677ebfe10d115e7faddd914b169ff5d6921b41ba0b7b4c31cc8ee39fe1f91b"} Sep 11 18:33:49.941504 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:49.941487 2561 scope.go:117] "RemoveContainer" containerID="64677ebfe10d115e7faddd914b169ff5d6921b41ba0b7b4c31cc8ee39fe1f91b" Sep 11 18:33:49.942688 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:49.942658 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:33:49.942739 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:49.942727 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:33:49.942775 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:49.942760 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:33:50.958286 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:50.958245 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc" event={"ID":"b79f3a4c-779f-4157-9746-bfe9348292b3","Type":"ContainerStarted","Data":"7be3b41565610a3e0e08e86292685ae57342ef764f0690126b1d15cf1f216ad2"} Sep 11 18:33:51.355798 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:51.355701 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc" Sep 11 18:33:51.356859 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:51.356823 2561 patch_prober.go:28] interesting pod/e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Sep 11 18:33:51.356986 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:51.356886 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc" podUID="b79f3a4c-779f-4157-9746-bfe9348292b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Sep 11 18:33:52.356673 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:52.356634 2561 patch_prober.go:28] interesting pod/e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Sep 11 18:33:52.357115 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:52.356704 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc" podUID="b79f3a4c-779f-4157-9746-bfe9348292b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Sep 11 18:33:52.702003 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:52.701954 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc" Sep 11 18:33:53.356684 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:53.356646 2561 patch_prober.go:28] interesting pod/e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Sep 11 18:33:53.357110 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:53.356705 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc" podUID="b79f3a4c-779f-4157-9746-bfe9348292b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Sep 11 18:33:54.356164 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:54.356127 2561 patch_prober.go:28] interesting pod/e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Sep 11 18:33:54.356409 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:54.356186 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc" podUID="b79f3a4c-779f-4157-9746-bfe9348292b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Sep 11 18:33:54.556069 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:54.556012 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7635-4f92c8"] Sep 11 18:33:54.556540 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:54.556523 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ab46e7b-3161-46a7-9603-629d75d31c1d" containerName="istio-proxy" Sep 11 18:33:54.556588 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:54.556541 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ab46e7b-3161-46a7-9603-629d75d31c1d" containerName="istio-proxy" Sep 11 18:33:54.556645 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:54.556636 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ab46e7b-3161-46a7-9603-629d75d31c1d" containerName="istio-proxy" Sep 11 18:33:54.556688 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:54.556646 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ab46e7b-3161-46a7-9603-629d75d31c1d" containerName="istio-proxy" Sep 11 18:33:54.558903 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:54.558839 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-4f92c8" Sep 11 18:33:54.564384 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:54.564351 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-bbqnn\"" Sep 11 18:33:54.568506 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:54.568471 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7635-4f92c8"] Sep 11 18:33:54.604785 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:54.604738 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw6sf\" (UniqueName: \"kubernetes.io/projected/e6b7af91-fe9c-4c87-b654-f2bb1d441bec-kube-api-access-fw6sf\") pod \"test-curl-7635-4f92c8\" (UID: \"e6b7af91-fe9c-4c87-b654-f2bb1d441bec\") " pod="opendatahub/test-curl-7635-4f92c8" Sep 11 18:33:54.707813 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:54.707756 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw6sf\" (UniqueName: \"kubernetes.io/projected/e6b7af91-fe9c-4c87-b654-f2bb1d441bec-kube-api-access-fw6sf\") pod \"test-curl-7635-4f92c8\" (UID: \"e6b7af91-fe9c-4c87-b654-f2bb1d441bec\") " pod="opendatahub/test-curl-7635-4f92c8" Sep 11 18:33:54.717303 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:54.717274 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw6sf\" (UniqueName: \"kubernetes.io/projected/e6b7af91-fe9c-4c87-b654-f2bb1d441bec-kube-api-access-fw6sf\") pod \"test-curl-7635-4f92c8\" (UID: \"e6b7af91-fe9c-4c87-b654-f2bb1d441bec\") " pod="opendatahub/test-curl-7635-4f92c8" Sep 11 18:33:54.875481 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:54.875449 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-4f92c8" Sep 11 18:33:55.174502 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:55.174475 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7635-4f92c8"] Sep 11 18:33:55.175736 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:33:55.175709 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode6b7af91_fe9c_4c87_b654_f2bb1d441bec.slice/crio-2e7283b08cabe205bdd4914f141f1b4e6a03cba530ef2bed4489e6082c9ebfdd WatchSource:0}: Error finding container 2e7283b08cabe205bdd4914f141f1b4e6a03cba530ef2bed4489e6082c9ebfdd: Status 404 returned error can't find the container with id 2e7283b08cabe205bdd4914f141f1b4e6a03cba530ef2bed4489e6082c9ebfdd Sep 11 18:33:55.178339 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:55.178321 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 18:33:55.356042 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:55.356003 2561 patch_prober.go:28] interesting pod/e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Sep 11 18:33:55.356240 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:55.356103 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc" podUID="b79f3a4c-779f-4157-9746-bfe9348292b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Sep 11 18:33:56.013673 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:56.013636 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7635-4f92c8" event={"ID":"e6b7af91-fe9c-4c87-b654-f2bb1d441bec","Type":"ContainerStarted","Data":"2e7283b08cabe205bdd4914f141f1b4e6a03cba530ef2bed4489e6082c9ebfdd"} Sep 11 18:33:56.361220 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:56.361137 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc" Sep 11 18:33:56.362436 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:56.362417 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc" Sep 11 18:33:57.022461 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:57.022422 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7635-4f92c8" event={"ID":"e6b7af91-fe9c-4c87-b654-f2bb1d441bec","Type":"ContainerStarted","Data":"c8b2360dd54e7e48683055a893cf52759ead6c2b214b1bbc3b06445f72ef6632"} Sep 11 18:33:57.038654 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:57.038603 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7635-4f92c8" podStartSLOduration=2.079236463 podStartE2EDuration="3.038586577s" totalImagesPullingTime="959.350114ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:33:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:33:57.036945952 +0000 UTC m=+1118.533963709" watchObservedRunningTime="2026-09-11 18:33:57.038586577 +0000 UTC m=+1118.535604371" Sep 11 18:33:57.626128 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:57.626091 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7635-4f92c8"] Sep 11 18:33:57.629379 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:57.629349 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7635-4f92c8"] Sep 11 18:33:57.741339 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:57.741264 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7635-4d23cd"] Sep 11 18:33:57.741729 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:57.741716 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ab46e7b-3161-46a7-9603-629d75d31c1d" containerName="istio-proxy" Sep 11 18:33:57.741772 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:57.741731 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ab46e7b-3161-46a7-9603-629d75d31c1d" containerName="istio-proxy" Sep 11 18:33:57.741807 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:57.741773 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6b7af91-fe9c-4c87-b654-f2bb1d441bec" containerName="test-curl-7635-4f92c8" Sep 11 18:33:57.741807 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:57.741779 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6b7af91-fe9c-4c87-b654-f2bb1d441bec" containerName="test-curl-7635-4f92c8" Sep 11 18:33:57.741875 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:57.741848 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6b7af91-fe9c-4c87-b654-f2bb1d441bec" containerName="test-curl-7635-4f92c8" Sep 11 18:33:57.744016 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:57.743998 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-4d23cd" Sep 11 18:33:57.751303 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:57.751276 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7635-4d23cd"] Sep 11 18:33:57.845786 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:57.845750 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n82dc\" (UniqueName: \"kubernetes.io/projected/51d67091-2062-4334-994f-09d836de0515-kube-api-access-n82dc\") pod \"test-curl-7635-4d23cd\" (UID: \"51d67091-2062-4334-994f-09d836de0515\") " pod="opendatahub/test-curl-7635-4d23cd" Sep 11 18:33:57.946519 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:57.946473 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n82dc\" (UniqueName: \"kubernetes.io/projected/51d67091-2062-4334-994f-09d836de0515-kube-api-access-n82dc\") pod \"test-curl-7635-4d23cd\" (UID: \"51d67091-2062-4334-994f-09d836de0515\") " pod="opendatahub/test-curl-7635-4d23cd" Sep 11 18:33:57.956308 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:57.956277 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n82dc\" (UniqueName: \"kubernetes.io/projected/51d67091-2062-4334-994f-09d836de0515-kube-api-access-n82dc\") pod \"test-curl-7635-4d23cd\" (UID: \"51d67091-2062-4334-994f-09d836de0515\") " pod="opendatahub/test-curl-7635-4d23cd" Sep 11 18:33:58.029914 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:33:58.029831 2561 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7635-4f92c8" container="test-curl-7635-4f92c8" Sep 11 18:33:58.055453 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:58.055428 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-4d23cd" Sep 11 18:33:58.331467 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:58.331434 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7635-4d23cd"] Sep 11 18:33:58.333394 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:33:58.333369 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod51d67091_2062_4334_994f_09d836de0515.slice/crio-63973c1f2a235fada390ba493a5c9bd7184dad90c6c6ce3d1e26c8640cbd3758 WatchSource:0}: Error finding container 63973c1f2a235fada390ba493a5c9bd7184dad90c6c6ce3d1e26c8640cbd3758: Status 404 returned error can't find the container with id 63973c1f2a235fada390ba493a5c9bd7184dad90c6c6ce3d1e26c8640cbd3758 Sep 11 18:33:59.038579 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:59.038490 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7635-4d23cd" event={"ID":"51d67091-2062-4334-994f-09d836de0515","Type":"ContainerStarted","Data":"01f6f6021f9684ee5d18a94619ccf37d958fda66f0219b246830d26e63da7887"} Sep 11 18:33:59.038579 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:59.038544 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7635-4d23cd" event={"ID":"51d67091-2062-4334-994f-09d836de0515","Type":"ContainerStarted","Data":"63973c1f2a235fada390ba493a5c9bd7184dad90c6c6ce3d1e26c8640cbd3758"} Sep 11 18:33:59.039025 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:59.038580 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7635-4f92c8" podUID="e6b7af91-fe9c-4c87-b654-f2bb1d441bec" containerName="test-curl-7635-4f92c8" containerID="cri-o://c8b2360dd54e7e48683055a893cf52759ead6c2b214b1bbc3b06445f72ef6632" gracePeriod=2 Sep 11 18:33:59.040660 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:33:59.040625 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7635-4f92c8\" is forbidden: User \"system:node:ip-10-0-138-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-128.ec2.internal' and this object" podUID="e6b7af91-fe9c-4c87-b654-f2bb1d441bec" pod="opendatahub/test-curl-7635-4f92c8" Sep 11 18:33:59.057396 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:33:59.057364 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7635-4f92c8\" is forbidden: User \"system:node:ip-10-0-138-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-128.ec2.internal' and this object" podUID="e6b7af91-fe9c-4c87-b654-f2bb1d441bec" pod="opendatahub/test-curl-7635-4f92c8" Sep 11 18:33:59.058581 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:59.058543 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7635-4d23cd" podStartSLOduration=2.058528832 podStartE2EDuration="2.058528832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:33:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:33:59.055579554 +0000 UTC m=+1120.552597309" watchObservedRunningTime="2026-09-11 18:33:59.058528832 +0000 UTC m=+1120.555546651" Sep 11 18:33:59.139646 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:33:59.139605 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7635-4f92c8\" is forbidden: User \"system:node:ip-10-0-138-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-128.ec2.internal' and this object" podUID="e6b7af91-fe9c-4c87-b654-f2bb1d441bec" pod="opendatahub/test-curl-7635-4f92c8" Sep 11 18:33:59.630334 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:59.630249 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7635-4d23cd"] Sep 11 18:33:59.633664 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:59.633631 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7635-4d23cd"] Sep 11 18:33:59.740999 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:59.740915 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7635-3157ab"] Sep 11 18:33:59.741496 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:59.741477 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51d67091-2062-4334-994f-09d836de0515" containerName="test-curl-7635-4d23cd" Sep 11 18:33:59.741559 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:59.741500 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="51d67091-2062-4334-994f-09d836de0515" containerName="test-curl-7635-4d23cd" Sep 11 18:33:59.741605 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:59.741586 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="51d67091-2062-4334-994f-09d836de0515" containerName="test-curl-7635-4d23cd" Sep 11 18:33:59.743651 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:59.743631 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-3157ab" Sep 11 18:33:59.748288 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:59.748260 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7635-3157ab"] Sep 11 18:33:59.868189 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:59.868152 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmrfh\" (UniqueName: \"kubernetes.io/projected/f8cea18e-d3c7-4624-be98-ceecf8e9f1aa-kube-api-access-mmrfh\") pod \"test-curl-7635-3157ab\" (UID: \"f8cea18e-d3c7-4624-be98-ceecf8e9f1aa\") " pod="opendatahub/test-curl-7635-3157ab" Sep 11 18:33:59.969518 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:59.969485 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmrfh\" (UniqueName: \"kubernetes.io/projected/f8cea18e-d3c7-4624-be98-ceecf8e9f1aa-kube-api-access-mmrfh\") pod \"test-curl-7635-3157ab\" (UID: \"f8cea18e-d3c7-4624-be98-ceecf8e9f1aa\") " pod="opendatahub/test-curl-7635-3157ab" Sep 11 18:33:59.978486 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:33:59.978464 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmrfh\" (UniqueName: \"kubernetes.io/projected/f8cea18e-d3c7-4624-be98-ceecf8e9f1aa-kube-api-access-mmrfh\") pod \"test-curl-7635-3157ab\" (UID: \"f8cea18e-d3c7-4624-be98-ceecf8e9f1aa\") " pod="opendatahub/test-curl-7635-3157ab" Sep 11 18:34:00.046135 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:34:00.046040 2561 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7635-4d23cd" container="test-curl-7635-4d23cd" Sep 11 18:34:00.055829 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:00.055805 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-3157ab" Sep 11 18:34:00.329484 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:00.329449 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7635-3157ab"] Sep 11 18:34:00.329864 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:34:00.329836 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf8cea18e_d3c7_4624_be98_ceecf8e9f1aa.slice/crio-9382db96516592b4be1a1b98cad70ddf612a6db20ef7c63a7747eb7f27232b89 WatchSource:0}: Error finding container 9382db96516592b4be1a1b98cad70ddf612a6db20ef7c63a7747eb7f27232b89: Status 404 returned error can't find the container with id 9382db96516592b4be1a1b98cad70ddf612a6db20ef7c63a7747eb7f27232b89 Sep 11 18:34:01.055551 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:01.055215 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7635-3157ab" event={"ID":"f8cea18e-d3c7-4624-be98-ceecf8e9f1aa","Type":"ContainerStarted","Data":"7df62023ac3d9d7ae415a7dc2725a5039c22c12341f7ffd933dc0eb2c75d610a"} Sep 11 18:34:01.055551 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:01.055264 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7635-3157ab" event={"ID":"f8cea18e-d3c7-4624-be98-ceecf8e9f1aa","Type":"ContainerStarted","Data":"9382db96516592b4be1a1b98cad70ddf612a6db20ef7c63a7747eb7f27232b89"} Sep 11 18:34:01.055551 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:01.055307 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7635-4d23cd" podUID="51d67091-2062-4334-994f-09d836de0515" containerName="test-curl-7635-4d23cd" containerID="cri-o://01f6f6021f9684ee5d18a94619ccf37d958fda66f0219b246830d26e63da7887" gracePeriod=2 Sep 11 18:34:01.071048 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:34:01.071012 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7635-4d23cd\" is forbidden: User \"system:node:ip-10-0-138-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-128.ec2.internal' and this object" podUID="51d67091-2062-4334-994f-09d836de0515" pod="opendatahub/test-curl-7635-4d23cd" Sep 11 18:34:01.071881 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:01.071832 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7635-3157ab" podStartSLOduration=2.071814868 podStartE2EDuration="2.071814868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:33:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:34:01.069313196 +0000 UTC m=+1122.566330952" watchObservedRunningTime="2026-09-11 18:34:01.071814868 +0000 UTC m=+1122.568832622" Sep 11 18:34:01.072664 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:34:01.072635 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7635-4d23cd\" is forbidden: User \"system:node:ip-10-0-138-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-128.ec2.internal' and this object" podUID="51d67091-2062-4334-994f-09d836de0515" pod="opendatahub/test-curl-7635-4d23cd" Sep 11 18:34:01.191159 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:01.191132 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-4f92c8" Sep 11 18:34:01.282735 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:01.282694 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fw6sf\" (UniqueName: \"kubernetes.io/projected/e6b7af91-fe9c-4c87-b654-f2bb1d441bec-kube-api-access-fw6sf\") pod \"e6b7af91-fe9c-4c87-b654-f2bb1d441bec\" (UID: \"e6b7af91-fe9c-4c87-b654-f2bb1d441bec\") " Sep 11 18:34:01.287380 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:01.287335 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6b7af91-fe9c-4c87-b654-f2bb1d441bec-kube-api-access-fw6sf" (OuterVolumeSpecName: "kube-api-access-fw6sf") pod "e6b7af91-fe9c-4c87-b654-f2bb1d441bec" (UID: "e6b7af91-fe9c-4c87-b654-f2bb1d441bec"). InnerVolumeSpecName "kube-api-access-fw6sf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:34:01.384761 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:01.384721 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw6sf\" (UniqueName: \"kubernetes.io/projected/e6b7af91-fe9c-4c87-b654-f2bb1d441bec-kube-api-access-fw6sf\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:34:01.653604 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:01.653575 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7635-3157ab"] Sep 11 18:34:01.659406 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:01.659373 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7635-3157ab"] Sep 11 18:34:01.768161 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:01.768123 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7635-71ba48"] Sep 11 18:34:01.768649 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:01.768632 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8cea18e-d3c7-4624-be98-ceecf8e9f1aa" containerName="test-curl-7635-3157ab" Sep 11 18:34:01.768695 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:01.768650 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8cea18e-d3c7-4624-be98-ceecf8e9f1aa" containerName="test-curl-7635-3157ab" Sep 11 18:34:01.768747 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:01.768737 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8cea18e-d3c7-4624-be98-ceecf8e9f1aa" containerName="test-curl-7635-3157ab" Sep 11 18:34:01.770666 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:01.770650 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-71ba48" Sep 11 18:34:01.778412 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:01.778220 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7635-71ba48"] Sep 11 18:34:01.789297 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:01.789258 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbntp\" (UniqueName: \"kubernetes.io/projected/7cccf31b-b0de-4862-8a39-67da98d2de02-kube-api-access-fbntp\") pod \"test-curl-7635-71ba48\" (UID: \"7cccf31b-b0de-4862-8a39-67da98d2de02\") " pod="opendatahub/test-curl-7635-71ba48" Sep 11 18:34:01.890367 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:01.890330 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbntp\" (UniqueName: \"kubernetes.io/projected/7cccf31b-b0de-4862-8a39-67da98d2de02-kube-api-access-fbntp\") pod \"test-curl-7635-71ba48\" (UID: \"7cccf31b-b0de-4862-8a39-67da98d2de02\") " pod="opendatahub/test-curl-7635-71ba48" Sep 11 18:34:01.901020 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:01.900986 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbntp\" (UniqueName: \"kubernetes.io/projected/7cccf31b-b0de-4862-8a39-67da98d2de02-kube-api-access-fbntp\") pod \"test-curl-7635-71ba48\" (UID: \"7cccf31b-b0de-4862-8a39-67da98d2de02\") " pod="opendatahub/test-curl-7635-71ba48" Sep 11 18:34:02.063887 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:02.063796 2561 generic.go:356] "Generic (PLEG): container finished" podID="e6b7af91-fe9c-4c87-b654-f2bb1d441bec" containerID="c8b2360dd54e7e48683055a893cf52759ead6c2b214b1bbc3b06445f72ef6632" exitCode=137 Sep 11 18:34:02.063887 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:02.063833 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-4f92c8" Sep 11 18:34:02.064359 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:02.063889 2561 scope.go:117] "RemoveContainer" containerID="c8b2360dd54e7e48683055a893cf52759ead6c2b214b1bbc3b06445f72ef6632" Sep 11 18:34:02.064359 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:34:02.064224 2561 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7635-3157ab" container="test-curl-7635-3157ab" Sep 11 18:34:02.074602 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:34:02.074493 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7635-4f92c8\" is forbidden: User \"system:node:ip-10-0-138-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-128.ec2.internal' and this object" podUID="e6b7af91-fe9c-4c87-b654-f2bb1d441bec" pod="opendatahub/test-curl-7635-4f92c8" Sep 11 18:34:02.077243 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:02.077222 2561 scope.go:117] "RemoveContainer" containerID="c8b2360dd54e7e48683055a893cf52759ead6c2b214b1bbc3b06445f72ef6632" Sep 11 18:34:02.077724 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:34:02.077705 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8b2360dd54e7e48683055a893cf52759ead6c2b214b1bbc3b06445f72ef6632\": container with ID starting with c8b2360dd54e7e48683055a893cf52759ead6c2b214b1bbc3b06445f72ef6632 not found: ID does not exist" containerID="c8b2360dd54e7e48683055a893cf52759ead6c2b214b1bbc3b06445f72ef6632" Sep 11 18:34:02.077795 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:02.077734 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8b2360dd54e7e48683055a893cf52759ead6c2b214b1bbc3b06445f72ef6632"} err="failed to get container status \"c8b2360dd54e7e48683055a893cf52759ead6c2b214b1bbc3b06445f72ef6632\": rpc error: code = NotFound desc = could not find container \"c8b2360dd54e7e48683055a893cf52759ead6c2b214b1bbc3b06445f72ef6632\": container with ID starting with c8b2360dd54e7e48683055a893cf52759ead6c2b214b1bbc3b06445f72ef6632 not found: ID does not exist" Sep 11 18:34:02.081895 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:02.081874 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-71ba48" Sep 11 18:34:02.353109 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:02.353079 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7635-71ba48"] Sep 11 18:34:02.354607 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:34:02.354583 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7cccf31b_b0de_4862_8a39_67da98d2de02.slice/crio-2d6a2c5fe0b3f18a2bd6680fc0e5d0470e7a2bd5043c58d37ee3a9d84c2c391c WatchSource:0}: Error finding container 2d6a2c5fe0b3f18a2bd6680fc0e5d0470e7a2bd5043c58d37ee3a9d84c2c391c: Status 404 returned error can't find the container with id 2d6a2c5fe0b3f18a2bd6680fc0e5d0470e7a2bd5043c58d37ee3a9d84c2c391c Sep 11 18:34:03.074719 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:03.074683 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7635-71ba48" event={"ID":"7cccf31b-b0de-4862-8a39-67da98d2de02","Type":"ContainerStarted","Data":"2ade42e5080552446e0b11a5cfdad7d92ed168799c6085f1571c265a3cd07ca2"} Sep 11 18:34:03.075079 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:03.074740 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7635-71ba48" event={"ID":"7cccf31b-b0de-4862-8a39-67da98d2de02","Type":"ContainerStarted","Data":"2d6a2c5fe0b3f18a2bd6680fc0e5d0470e7a2bd5043c58d37ee3a9d84c2c391c"} Sep 11 18:34:03.075079 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:03.074778 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7635-3157ab" podUID="f8cea18e-d3c7-4624-be98-ceecf8e9f1aa" containerName="test-curl-7635-3157ab" containerID="cri-o://7df62023ac3d9d7ae415a7dc2725a5039c22c12341f7ffd933dc0eb2c75d610a" gracePeriod=2 Sep 11 18:34:03.090198 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:34:03.090158 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7635-4f92c8\" is forbidden: User \"system:node:ip-10-0-138-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-128.ec2.internal' and this object" podUID="e6b7af91-fe9c-4c87-b654-f2bb1d441bec" pod="opendatahub/test-curl-7635-4f92c8" Sep 11 18:34:03.091143 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:03.091103 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7635-71ba48" podStartSLOduration=2.091089294 podStartE2EDuration="2.091089294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:34:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:34:03.0885899 +0000 UTC m=+1124.585607655" watchObservedRunningTime="2026-09-11 18:34:03.091089294 +0000 UTC m=+1124.588107045" Sep 11 18:34:03.091661 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:34:03.091628 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7635-3157ab\" is forbidden: User \"system:node:ip-10-0-138-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-128.ec2.internal' and this object" podUID="f8cea18e-d3c7-4624-be98-ceecf8e9f1aa" pod="opendatahub/test-curl-7635-3157ab" Sep 11 18:34:03.093218 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:34:03.093191 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7635-4f92c8\" is forbidden: User \"system:node:ip-10-0-138-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-128.ec2.internal' and this object" podUID="e6b7af91-fe9c-4c87-b654-f2bb1d441bec" pod="opendatahub/test-curl-7635-4f92c8" Sep 11 18:34:03.094772 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:34:03.094702 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7635-3157ab\" is forbidden: User \"system:node:ip-10-0-138-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-128.ec2.internal' and this object" podUID="f8cea18e-d3c7-4624-be98-ceecf8e9f1aa" pod="opendatahub/test-curl-7635-3157ab" Sep 11 18:34:03.141105 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:03.141074 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6b7af91-fe9c-4c87-b654-f2bb1d441bec" path="/var/lib/kubelet/pods/e6b7af91-fe9c-4c87-b654-f2bb1d441bec/volumes" Sep 11 18:34:03.245354 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:03.245329 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-4d23cd" Sep 11 18:34:03.306111 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:03.306075 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n82dc\" (UniqueName: \"kubernetes.io/projected/51d67091-2062-4334-994f-09d836de0515-kube-api-access-n82dc\") pod \"51d67091-2062-4334-994f-09d836de0515\" (UID: \"51d67091-2062-4334-994f-09d836de0515\") " Sep 11 18:34:03.307875 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:03.307849 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51d67091-2062-4334-994f-09d836de0515-kube-api-access-n82dc" (OuterVolumeSpecName: "kube-api-access-n82dc") pod "51d67091-2062-4334-994f-09d836de0515" (UID: "51d67091-2062-4334-994f-09d836de0515"). InnerVolumeSpecName "kube-api-access-n82dc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:34:03.407200 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:03.407164 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n82dc\" (UniqueName: \"kubernetes.io/projected/51d67091-2062-4334-994f-09d836de0515-kube-api-access-n82dc\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:34:03.652181 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:03.652097 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7635-71ba48"] Sep 11 18:34:03.658620 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:03.658592 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7635-71ba48"] Sep 11 18:34:04.083302 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:04.083260 2561 generic.go:356] "Generic (PLEG): container finished" podID="51d67091-2062-4334-994f-09d836de0515" containerID="01f6f6021f9684ee5d18a94619ccf37d958fda66f0219b246830d26e63da7887" exitCode=137 Sep 11 18:34:04.083302 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:04.083307 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-4d23cd" Sep 11 18:34:04.083847 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:04.083383 2561 scope.go:117] "RemoveContainer" containerID="01f6f6021f9684ee5d18a94619ccf37d958fda66f0219b246830d26e63da7887" Sep 11 18:34:04.083847 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:34:04.083608 2561 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7635-71ba48" container="test-curl-7635-71ba48" Sep 11 18:34:04.094996 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:34:04.094965 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7635-4d23cd\" is forbidden: User \"system:node:ip-10-0-138-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-128.ec2.internal' and this object" podUID="51d67091-2062-4334-994f-09d836de0515" pod="opendatahub/test-curl-7635-4d23cd" Sep 11 18:34:04.095250 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:04.095233 2561 scope.go:117] "RemoveContainer" containerID="01f6f6021f9684ee5d18a94619ccf37d958fda66f0219b246830d26e63da7887" Sep 11 18:34:04.095567 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:34:04.095545 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01f6f6021f9684ee5d18a94619ccf37d958fda66f0219b246830d26e63da7887\": container with ID starting with 01f6f6021f9684ee5d18a94619ccf37d958fda66f0219b246830d26e63da7887 not found: ID does not exist" containerID="01f6f6021f9684ee5d18a94619ccf37d958fda66f0219b246830d26e63da7887" Sep 11 18:34:04.095639 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:04.095580 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01f6f6021f9684ee5d18a94619ccf37d958fda66f0219b246830d26e63da7887"} err="failed to get container status \"01f6f6021f9684ee5d18a94619ccf37d958fda66f0219b246830d26e63da7887\": rpc error: code = NotFound desc = could not find container \"01f6f6021f9684ee5d18a94619ccf37d958fda66f0219b246830d26e63da7887\": container with ID starting with 01f6f6021f9684ee5d18a94619ccf37d958fda66f0219b246830d26e63da7887 not found: ID does not exist" Sep 11 18:34:04.511602 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:04.511557 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj"] Sep 11 18:34:04.512109 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:04.512090 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cccf31b-b0de-4862-8a39-67da98d2de02" containerName="test-curl-7635-71ba48" Sep 11 18:34:04.512214 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:04.512111 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cccf31b-b0de-4862-8a39-67da98d2de02" containerName="test-curl-7635-71ba48" Sep 11 18:34:04.512283 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:04.512269 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cccf31b-b0de-4862-8a39-67da98d2de02" containerName="test-curl-7635-71ba48" Sep 11 18:34:04.514885 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:04.514863 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj" Sep 11 18:34:04.516866 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:34:04.516839 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7635-4d23cd\" is forbidden: User \"system:node:ip-10-0-138-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-128.ec2.internal' and this object" podUID="51d67091-2062-4334-994f-09d836de0515" pod="opendatahub/test-curl-7635-4d23cd" Sep 11 18:34:04.517140 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:04.517118 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-18a45f6a-openshift-default-dockercfg-4tbz9\"" Sep 11 18:34:04.526898 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:04.526873 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj"] Sep 11 18:34:04.620233 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:04.619658 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ae85a40-303c-45f3-9eaf-72b042756d2e-istio-envoy\") pod \"e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj\" (UID: \"0ae85a40-303c-45f3-9eaf-72b042756d2e\") " pod="openshift-ingress/e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj" Sep 11 18:34:04.620233 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:04.619726 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ae85a40-303c-45f3-9eaf-72b042756d2e-istio-data\") pod \"e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj\" (UID: \"0ae85a40-303c-45f3-9eaf-72b042756d2e\") " pod="openshift-ingress/e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj" Sep 11 18:34:04.620233 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:04.619758 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7jcw\" (UniqueName: \"kubernetes.io/projected/0ae85a40-303c-45f3-9eaf-72b042756d2e-kube-api-access-d7jcw\") pod \"e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj\" (UID: \"0ae85a40-303c-45f3-9eaf-72b042756d2e\") " pod="openshift-ingress/e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj" Sep 11 18:34:04.620233 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:04.619793 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ae85a40-303c-45f3-9eaf-72b042756d2e-workload-certs\") pod \"e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj\" (UID: \"0ae85a40-303c-45f3-9eaf-72b042756d2e\") " pod="openshift-ingress/e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj" Sep 11 18:34:04.620233 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:04.619841 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ae85a40-303c-45f3-9eaf-72b042756d2e-workload-socket\") pod \"e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj\" (UID: \"0ae85a40-303c-45f3-9eaf-72b042756d2e\") " pod="openshift-ingress/e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj" Sep 11 18:34:04.620233 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:04.619930 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ae85a40-303c-45f3-9eaf-72b042756d2e-istiod-ca-cert\") pod \"e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj\" (UID: \"0ae85a40-303c-45f3-9eaf-72b042756d2e\") " pod="openshift-ingress/e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj" Sep 11 18:34:04.620233 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:04.619970 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ae85a40-303c-45f3-9eaf-72b042756d2e-istio-token\") pod \"e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj\" (UID: \"0ae85a40-303c-45f3-9eaf-72b042756d2e\") " pod="openshift-ingress/e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj" Sep 11 18:34:04.620233 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:04.620037 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ae85a40-303c-45f3-9eaf-72b042756d2e-istio-podinfo\") pod \"e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj\" (UID: \"0ae85a40-303c-45f3-9eaf-72b042756d2e\") " pod="openshift-ingress/e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj" Sep 11 18:34:04.620233 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:04.620085 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ae85a40-303c-45f3-9eaf-72b042756d2e-credential-socket\") pod \"e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj\" (UID: \"0ae85a40-303c-45f3-9eaf-72b042756d2e\") " pod="openshift-ingress/e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj" Sep 11 18:34:04.721600 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:04.721567 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ae85a40-303c-45f3-9eaf-72b042756d2e-istio-podinfo\") pod \"e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj\" (UID: \"0ae85a40-303c-45f3-9eaf-72b042756d2e\") " pod="openshift-ingress/e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj" Sep 11 18:34:04.721802 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:04.721613 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ae85a40-303c-45f3-9eaf-72b042756d2e-credential-socket\") pod \"e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj\" (UID: \"0ae85a40-303c-45f3-9eaf-72b042756d2e\") " pod="openshift-ingress/e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj" Sep 11 18:34:04.721802 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:04.721660 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ae85a40-303c-45f3-9eaf-72b042756d2e-istio-envoy\") pod \"e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj\" (UID: \"0ae85a40-303c-45f3-9eaf-72b042756d2e\") " pod="openshift-ingress/e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj" Sep 11 18:34:04.721802 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:04.721694 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ae85a40-303c-45f3-9eaf-72b042756d2e-istio-data\") pod \"e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj\" (UID: \"0ae85a40-303c-45f3-9eaf-72b042756d2e\") " pod="openshift-ingress/e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj" Sep 11 18:34:04.721802 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:04.721714 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7jcw\" (UniqueName: \"kubernetes.io/projected/0ae85a40-303c-45f3-9eaf-72b042756d2e-kube-api-access-d7jcw\") pod \"e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj\" (UID: \"0ae85a40-303c-45f3-9eaf-72b042756d2e\") " pod="openshift-ingress/e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj" Sep 11 18:34:04.721802 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:04.721734 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ae85a40-303c-45f3-9eaf-72b042756d2e-workload-certs\") pod \"e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj\" (UID: \"0ae85a40-303c-45f3-9eaf-72b042756d2e\") " pod="openshift-ingress/e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj" Sep 11 18:34:04.721802 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:04.721776 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ae85a40-303c-45f3-9eaf-72b042756d2e-workload-socket\") pod \"e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj\" (UID: \"0ae85a40-303c-45f3-9eaf-72b042756d2e\") " pod="openshift-ingress/e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj" Sep 11 18:34:04.722173 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:04.721829 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ae85a40-303c-45f3-9eaf-72b042756d2e-istiod-ca-cert\") pod \"e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj\" (UID: \"0ae85a40-303c-45f3-9eaf-72b042756d2e\") " pod="openshift-ingress/e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj" Sep 11 18:34:04.722173 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:04.721867 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ae85a40-303c-45f3-9eaf-72b042756d2e-istio-token\") pod \"e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj\" (UID: \"0ae85a40-303c-45f3-9eaf-72b042756d2e\") " pod="openshift-ingress/e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj" Sep 11 18:34:04.722286 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:04.722250 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ae85a40-303c-45f3-9eaf-72b042756d2e-workload-certs\") pod \"e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj\" (UID: \"0ae85a40-303c-45f3-9eaf-72b042756d2e\") " pod="openshift-ingress/e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj" Sep 11 18:34:04.722366 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:04.722336 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ae85a40-303c-45f3-9eaf-72b042756d2e-credential-socket\") pod \"e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj\" (UID: \"0ae85a40-303c-45f3-9eaf-72b042756d2e\") " pod="openshift-ingress/e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj" Sep 11 18:34:04.722782 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:04.722387 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ae85a40-303c-45f3-9eaf-72b042756d2e-istio-data\") pod \"e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj\" (UID: \"0ae85a40-303c-45f3-9eaf-72b042756d2e\") " pod="openshift-ingress/e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj" Sep 11 18:34:04.722782 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:04.722561 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ae85a40-303c-45f3-9eaf-72b042756d2e-workload-socket\") pod \"e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj\" (UID: \"0ae85a40-303c-45f3-9eaf-72b042756d2e\") " pod="openshift-ingress/e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj" Sep 11 18:34:04.722863 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:04.722814 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ae85a40-303c-45f3-9eaf-72b042756d2e-istiod-ca-cert\") pod \"e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj\" (UID: \"0ae85a40-303c-45f3-9eaf-72b042756d2e\") " pod="openshift-ingress/e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj" Sep 11 18:34:04.724101 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:04.724039 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ae85a40-303c-45f3-9eaf-72b042756d2e-istio-envoy\") pod \"e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj\" (UID: \"0ae85a40-303c-45f3-9eaf-72b042756d2e\") " pod="openshift-ingress/e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj" Sep 11 18:34:04.724338 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:04.724322 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ae85a40-303c-45f3-9eaf-72b042756d2e-istio-podinfo\") pod \"e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj\" (UID: \"0ae85a40-303c-45f3-9eaf-72b042756d2e\") " pod="openshift-ingress/e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj" Sep 11 18:34:04.730930 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:04.730901 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ae85a40-303c-45f3-9eaf-72b042756d2e-istio-token\") pod \"e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj\" (UID: \"0ae85a40-303c-45f3-9eaf-72b042756d2e\") " pod="openshift-ingress/e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj" Sep 11 18:34:04.731015 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:04.730967 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7jcw\" (UniqueName: \"kubernetes.io/projected/0ae85a40-303c-45f3-9eaf-72b042756d2e-kube-api-access-d7jcw\") pod \"e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj\" (UID: \"0ae85a40-303c-45f3-9eaf-72b042756d2e\") " pod="openshift-ingress/e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj" Sep 11 18:34:04.829272 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:04.829184 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj" Sep 11 18:34:05.094226 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:05.094195 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7635-71ba48" podUID="7cccf31b-b0de-4862-8a39-67da98d2de02" containerName="test-curl-7635-71ba48" containerID="cri-o://2ade42e5080552446e0b11a5cfdad7d92ed168799c6085f1571c265a3cd07ca2" gracePeriod=2 Sep 11 18:34:05.096363 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:34:05.096332 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7635-71ba48\" is forbidden: User \"system:node:ip-10-0-138-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-128.ec2.internal' and this object" podUID="7cccf31b-b0de-4862-8a39-67da98d2de02" pod="opendatahub/test-curl-7635-71ba48" Sep 11 18:34:05.098186 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:34:05.098157 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7635-4d23cd\" is forbidden: User \"system:node:ip-10-0-138-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-128.ec2.internal' and this object" podUID="51d67091-2062-4334-994f-09d836de0515" pod="opendatahub/test-curl-7635-4d23cd" Sep 11 18:34:05.114205 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:05.114181 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj"] Sep 11 18:34:05.128410 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:34:05.128386 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ae85a40_303c_45f3_9eaf_72b042756d2e.slice/crio-0fd98b86211f19ca25d221c858378d30618f23bb9beb47cd3d8b2f566e351eed WatchSource:0}: Error finding container 0fd98b86211f19ca25d221c858378d30618f23bb9beb47cd3d8b2f566e351eed: Status 404 returned error can't find the container with id 0fd98b86211f19ca25d221c858378d30618f23bb9beb47cd3d8b2f566e351eed Sep 11 18:34:05.130792 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:05.130760 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:34:05.130884 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:05.130840 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:34:05.130944 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:05.130886 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:34:05.140283 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:05.140259 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51d67091-2062-4334-994f-09d836de0515" path="/var/lib/kubelet/pods/51d67091-2062-4334-994f-09d836de0515/volumes" Sep 11 18:34:05.255777 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:05.255750 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-3157ab" Sep 11 18:34:05.327942 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:05.327905 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmrfh\" (UniqueName: \"kubernetes.io/projected/f8cea18e-d3c7-4624-be98-ceecf8e9f1aa-kube-api-access-mmrfh\") pod \"f8cea18e-d3c7-4624-be98-ceecf8e9f1aa\" (UID: \"f8cea18e-d3c7-4624-be98-ceecf8e9f1aa\") " Sep 11 18:34:05.330037 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:05.329975 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8cea18e-d3c7-4624-be98-ceecf8e9f1aa-kube-api-access-mmrfh" (OuterVolumeSpecName: "kube-api-access-mmrfh") pod "f8cea18e-d3c7-4624-be98-ceecf8e9f1aa" (UID: "f8cea18e-d3c7-4624-be98-ceecf8e9f1aa"). InnerVolumeSpecName "kube-api-access-mmrfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:34:05.429865 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:05.429826 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmrfh\" (UniqueName: \"kubernetes.io/projected/f8cea18e-d3c7-4624-be98-ceecf8e9f1aa-kube-api-access-mmrfh\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:34:06.103195 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:06.103101 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj" event={"ID":"0ae85a40-303c-45f3-9eaf-72b042756d2e","Type":"ContainerStarted","Data":"401892651b30cf81a0fcf502e6949f755c83625b1ca16db5887caf44496d62ef"} Sep 11 18:34:06.103615 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:06.103213 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj" event={"ID":"0ae85a40-303c-45f3-9eaf-72b042756d2e","Type":"ContainerStarted","Data":"0fd98b86211f19ca25d221c858378d30618f23bb9beb47cd3d8b2f566e351eed"} Sep 11 18:34:06.104628 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:06.104597 2561 generic.go:356] "Generic (PLEG): container finished" podID="f8cea18e-d3c7-4624-be98-ceecf8e9f1aa" containerID="7df62023ac3d9d7ae415a7dc2725a5039c22c12341f7ffd933dc0eb2c75d610a" exitCode=137 Sep 11 18:34:06.104738 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:06.104648 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-3157ab" Sep 11 18:34:06.104738 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:06.104699 2561 scope.go:117] "RemoveContainer" containerID="7df62023ac3d9d7ae415a7dc2725a5039c22c12341f7ffd933dc0eb2c75d610a" Sep 11 18:34:06.115040 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:06.115022 2561 scope.go:117] "RemoveContainer" containerID="7df62023ac3d9d7ae415a7dc2725a5039c22c12341f7ffd933dc0eb2c75d610a" Sep 11 18:34:06.115334 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:34:06.115312 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7df62023ac3d9d7ae415a7dc2725a5039c22c12341f7ffd933dc0eb2c75d610a\": container with ID starting with 7df62023ac3d9d7ae415a7dc2725a5039c22c12341f7ffd933dc0eb2c75d610a not found: ID does not exist" containerID="7df62023ac3d9d7ae415a7dc2725a5039c22c12341f7ffd933dc0eb2c75d610a" Sep 11 18:34:06.115394 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:06.115346 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7df62023ac3d9d7ae415a7dc2725a5039c22c12341f7ffd933dc0eb2c75d610a"} err="failed to get container status \"7df62023ac3d9d7ae415a7dc2725a5039c22c12341f7ffd933dc0eb2c75d610a\": rpc error: code = NotFound desc = could not find container \"7df62023ac3d9d7ae415a7dc2725a5039c22c12341f7ffd933dc0eb2c75d610a\": container with ID starting with 7df62023ac3d9d7ae415a7dc2725a5039c22c12341f7ffd933dc0eb2c75d610a not found: ID does not exist" Sep 11 18:34:06.124699 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:06.124655 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj" podStartSLOduration=2.1246424 podStartE2EDuration="2.1246424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:34:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:34:06.123229674 +0000 UTC m=+1127.620247450" watchObservedRunningTime="2026-09-11 18:34:06.1246424 +0000 UTC m=+1127.621660153" Sep 11 18:34:06.125248 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:34:06.125223 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7635-3157ab\" is forbidden: User \"system:node:ip-10-0-138-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-128.ec2.internal' and this object" podUID="f8cea18e-d3c7-4624-be98-ceecf8e9f1aa" pod="opendatahub/test-curl-7635-3157ab" Sep 11 18:34:06.830037 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:06.829999 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj" Sep 11 18:34:06.835087 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:06.835049 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj" Sep 11 18:34:06.837197 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:34:06.837168 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7635-3157ab\" is forbidden: User \"system:node:ip-10-0-138-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-128.ec2.internal' and this object" podUID="f8cea18e-d3c7-4624-be98-ceecf8e9f1aa" pod="opendatahub/test-curl-7635-3157ab" Sep 11 18:34:07.114314 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:07.114293 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj" Sep 11 18:34:07.115170 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:07.115150 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj" Sep 11 18:34:07.138090 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:34:07.138030 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7635-3157ab\" is forbidden: User \"system:node:ip-10-0-138-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-128.ec2.internal' and this object" podUID="f8cea18e-d3c7-4624-be98-ceecf8e9f1aa" pod="opendatahub/test-curl-7635-3157ab" Sep 11 18:34:07.141820 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:07.141791 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8cea18e-d3c7-4624-be98-ceecf8e9f1aa" path="/var/lib/kubelet/pods/f8cea18e-d3c7-4624-be98-ceecf8e9f1aa/volumes" Sep 11 18:34:07.280488 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:07.280453 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-71ba48" Sep 11 18:34:07.356466 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:07.356370 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fbntp\" (UniqueName: \"kubernetes.io/projected/7cccf31b-b0de-4862-8a39-67da98d2de02-kube-api-access-fbntp\") pod \"7cccf31b-b0de-4862-8a39-67da98d2de02\" (UID: \"7cccf31b-b0de-4862-8a39-67da98d2de02\") " Sep 11 18:34:07.358427 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:07.358399 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cccf31b-b0de-4862-8a39-67da98d2de02-kube-api-access-fbntp" (OuterVolumeSpecName: "kube-api-access-fbntp") pod "7cccf31b-b0de-4862-8a39-67da98d2de02" (UID: "7cccf31b-b0de-4862-8a39-67da98d2de02"). InnerVolumeSpecName "kube-api-access-fbntp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:34:07.458152 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:07.458114 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbntp\" (UniqueName: \"kubernetes.io/projected/7cccf31b-b0de-4862-8a39-67da98d2de02-kube-api-access-fbntp\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:34:08.122493 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:08.122463 2561 generic.go:356] "Generic (PLEG): container finished" podID="7cccf31b-b0de-4862-8a39-67da98d2de02" containerID="2ade42e5080552446e0b11a5cfdad7d92ed168799c6085f1571c265a3cd07ca2" exitCode=137 Sep 11 18:34:08.122493 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:08.122504 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7635-71ba48" Sep 11 18:34:08.123003 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:08.122564 2561 scope.go:117] "RemoveContainer" containerID="2ade42e5080552446e0b11a5cfdad7d92ed168799c6085f1571c265a3cd07ca2" Sep 11 18:34:08.132733 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:08.132576 2561 scope.go:117] "RemoveContainer" containerID="2ade42e5080552446e0b11a5cfdad7d92ed168799c6085f1571c265a3cd07ca2" Sep 11 18:34:08.132901 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:34:08.132883 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ade42e5080552446e0b11a5cfdad7d92ed168799c6085f1571c265a3cd07ca2\": container with ID starting with 2ade42e5080552446e0b11a5cfdad7d92ed168799c6085f1571c265a3cd07ca2 not found: ID does not exist" containerID="2ade42e5080552446e0b11a5cfdad7d92ed168799c6085f1571c265a3cd07ca2" Sep 11 18:34:08.132941 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:08.132908 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ade42e5080552446e0b11a5cfdad7d92ed168799c6085f1571c265a3cd07ca2"} err="failed to get container status \"2ade42e5080552446e0b11a5cfdad7d92ed168799c6085f1571c265a3cd07ca2\": rpc error: code = NotFound desc = could not find container \"2ade42e5080552446e0b11a5cfdad7d92ed168799c6085f1571c265a3cd07ca2\": container with ID starting with 2ade42e5080552446e0b11a5cfdad7d92ed168799c6085f1571c265a3cd07ca2 not found: ID does not exist" Sep 11 18:34:08.133187 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:34:08.133165 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7635-71ba48\" is forbidden: User \"system:node:ip-10-0-138-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-128.ec2.internal' and this object" podUID="7cccf31b-b0de-4862-8a39-67da98d2de02" pod="opendatahub/test-curl-7635-71ba48" Sep 11 18:34:09.138017 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:34:09.137983 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7635-71ba48\" is forbidden: User \"system:node:ip-10-0-138-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-128.ec2.internal' and this object" podUID="7cccf31b-b0de-4862-8a39-67da98d2de02" pod="opendatahub/test-curl-7635-71ba48" Sep 11 18:34:09.139888 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:09.139863 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cccf31b-b0de-4862-8a39-67da98d2de02" path="/var/lib/kubelet/pods/7cccf31b-b0de-4862-8a39-67da98d2de02/volumes" Sep 11 18:34:14.087243 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:14.087206 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-18a45f6a-596dcbb4cd-zjpgq"] Sep 11 18:34:14.090480 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:14.090459 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-18a45f6a-596dcbb4cd-zjpgq" Sep 11 18:34:14.092762 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:14.092740 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-18a45f6a-dockercfg-566tj\"" Sep 11 18:34:14.092871 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:14.092794 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-18a45f6a\"" Sep 11 18:34:14.102897 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:14.102872 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-18a45f6a-596dcbb4cd-zjpgq"] Sep 11 18:34:14.233689 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:14.233659 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qfm6\" (UniqueName: \"kubernetes.io/projected/05e89f9e-a6a1-4652-b69d-6554cea2e82d-kube-api-access-7qfm6\") pod \"payload-pre-processing-e2e-mt-18a45f6a-596dcbb4cd-zjpgq\" (UID: \"05e89f9e-a6a1-4652-b69d-6554cea2e82d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-18a45f6a-596dcbb4cd-zjpgq" Sep 11 18:34:14.233876 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:14.233740 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05e89f9e-a6a1-4652-b69d-6554cea2e82d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-18a45f6a-596dcbb4cd-zjpgq\" (UID: \"05e89f9e-a6a1-4652-b69d-6554cea2e82d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-18a45f6a-596dcbb4cd-zjpgq" Sep 11 18:34:14.335289 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:14.335252 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05e89f9e-a6a1-4652-b69d-6554cea2e82d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-18a45f6a-596dcbb4cd-zjpgq\" (UID: \"05e89f9e-a6a1-4652-b69d-6554cea2e82d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-18a45f6a-596dcbb4cd-zjpgq" Sep 11 18:34:14.335458 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:14.335370 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qfm6\" (UniqueName: \"kubernetes.io/projected/05e89f9e-a6a1-4652-b69d-6554cea2e82d-kube-api-access-7qfm6\") pod \"payload-pre-processing-e2e-mt-18a45f6a-596dcbb4cd-zjpgq\" (UID: \"05e89f9e-a6a1-4652-b69d-6554cea2e82d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-18a45f6a-596dcbb4cd-zjpgq" Sep 11 18:34:14.335963 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:14.335944 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05e89f9e-a6a1-4652-b69d-6554cea2e82d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-18a45f6a-596dcbb4cd-zjpgq\" (UID: \"05e89f9e-a6a1-4652-b69d-6554cea2e82d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-18a45f6a-596dcbb4cd-zjpgq" Sep 11 18:34:14.346109 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:14.346043 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qfm6\" (UniqueName: \"kubernetes.io/projected/05e89f9e-a6a1-4652-b69d-6554cea2e82d-kube-api-access-7qfm6\") pod \"payload-pre-processing-e2e-mt-18a45f6a-596dcbb4cd-zjpgq\" (UID: \"05e89f9e-a6a1-4652-b69d-6554cea2e82d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-18a45f6a-596dcbb4cd-zjpgq" Sep 11 18:34:14.402451 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:14.402414 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-18a45f6a-596dcbb4cd-zjpgq" Sep 11 18:34:14.679584 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:14.679549 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-18a45f6a-596dcbb4cd-zjpgq"] Sep 11 18:34:14.680932 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:34:14.680905 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05e89f9e_a6a1_4652_b69d_6554cea2e82d.slice/crio-e941f82201fc505f64afb3444d5f1b807dee0eaed6ceb799abdb82b35d992ad5 WatchSource:0}: Error finding container e941f82201fc505f64afb3444d5f1b807dee0eaed6ceb799abdb82b35d992ad5: Status 404 returned error can't find the container with id e941f82201fc505f64afb3444d5f1b807dee0eaed6ceb799abdb82b35d992ad5 Sep 11 18:34:15.189641 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:15.189599 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-18a45f6a-596dcbb4cd-zjpgq" event={"ID":"05e89f9e-a6a1-4652-b69d-6554cea2e82d","Type":"ContainerStarted","Data":"04d7d5e8421653c8af983d480815433d71af068f9047d85d18c7b5c733ed804a"} Sep 11 18:34:15.190428 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:15.189664 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-18a45f6a-596dcbb4cd-zjpgq" event={"ID":"05e89f9e-a6a1-4652-b69d-6554cea2e82d","Type":"ContainerStarted","Data":"e941f82201fc505f64afb3444d5f1b807dee0eaed6ceb799abdb82b35d992ad5"} Sep 11 18:34:15.190428 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:15.189750 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-18a45f6a-596dcbb4cd-zjpgq" Sep 11 18:34:15.207580 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:15.207480 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-18a45f6a-596dcbb4cd-zjpgq" podStartSLOduration=1.207460136 podStartE2EDuration="1.207460136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:34:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:34:15.207028878 +0000 UTC m=+1136.704046654" watchObservedRunningTime="2026-09-11 18:34:15.207460136 +0000 UTC m=+1136.704477890" Sep 11 18:34:19.456088 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:19.456034 2561 scope.go:117] "RemoveContainer" containerID="f003b2e5e71f5b3e8bd3ab161125cd97a4a0996c2def74989639c624c703c8f1" Sep 11 18:34:26.201107 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:26.201066 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-18a45f6a-596dcbb4cd-zjpgq" Sep 11 18:34:41.394014 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:41.393921 2561 generic.go:356] "Generic (PLEG): container finished" podID="0ae85a40-303c-45f3-9eaf-72b042756d2e" containerID="401892651b30cf81a0fcf502e6949f755c83625b1ca16db5887caf44496d62ef" exitCode=0 Sep 11 18:34:41.394014 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:41.393964 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj" event={"ID":"0ae85a40-303c-45f3-9eaf-72b042756d2e","Type":"ContainerDied","Data":"401892651b30cf81a0fcf502e6949f755c83625b1ca16db5887caf44496d62ef"} Sep 11 18:34:41.394595 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:41.394429 2561 scope.go:117] "RemoveContainer" containerID="401892651b30cf81a0fcf502e6949f755c83625b1ca16db5887caf44496d62ef" Sep 11 18:34:41.395719 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:41.395684 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:34:41.395805 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:41.395761 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:34:41.395805 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:41.395794 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:34:42.413865 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:42.413822 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj" event={"ID":"0ae85a40-303c-45f3-9eaf-72b042756d2e","Type":"ContainerStarted","Data":"4cfe11f0ce5b790cad2706ce6eba3e0a702d0cac1e73afa12d587488aee22392"} Sep 11 18:34:42.829619 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:42.829535 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj" Sep 11 18:34:42.829772 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:42.829737 2561 patch_prober.go:28] interesting pod/e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Sep 11 18:34:42.829815 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:42.829790 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj" podUID="0ae85a40-303c-45f3-9eaf-72b042756d2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Sep 11 18:34:43.829819 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:43.829775 2561 patch_prober.go:28] interesting pod/e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Sep 11 18:34:43.830233 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:43.829838 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj" podUID="0ae85a40-303c-45f3-9eaf-72b042756d2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Sep 11 18:34:44.829940 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:44.829899 2561 patch_prober.go:28] interesting pod/e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Sep 11 18:34:44.830367 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:44.829972 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj" podUID="0ae85a40-303c-45f3-9eaf-72b042756d2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Sep 11 18:34:45.830314 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:45.830270 2561 patch_prober.go:28] interesting pod/e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Sep 11 18:34:45.830721 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:45.830351 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj" podUID="0ae85a40-303c-45f3-9eaf-72b042756d2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Sep 11 18:34:46.829654 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:46.829618 2561 patch_prober.go:28] interesting pod/e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Sep 11 18:34:46.829838 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:46.829674 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj" podUID="0ae85a40-303c-45f3-9eaf-72b042756d2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Sep 11 18:34:47.833861 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:47.833827 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj" Sep 11 18:34:47.834284 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:47.834262 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj" Sep 11 18:34:47.834937 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:47.834921 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj" Sep 11 18:34:53.001975 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:53.001940 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d7509-2mjmq"] Sep 11 18:34:53.004724 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:53.004706 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d7509-2mjmq" Sep 11 18:34:53.007083 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:53.007038 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 18:34:53.007208 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:53.007195 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n968b\"" Sep 11 18:34:53.016896 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:53.016866 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d7509-2mjmq"] Sep 11 18:34:53.166948 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:53.166910 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b564ab7-13d0-4543-ab0a-c9d52f1f2c77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7d7509-2mjmq\" (UID: \"4b564ab7-13d0-4543-ab0a-c9d52f1f2c77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d7509-2mjmq" Sep 11 18:34:53.268417 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:53.268311 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b564ab7-13d0-4543-ab0a-c9d52f1f2c77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7d7509-2mjmq\" (UID: \"4b564ab7-13d0-4543-ab0a-c9d52f1f2c77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d7509-2mjmq" Sep 11 18:34:53.268981 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:53.268959 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b564ab7-13d0-4543-ab0a-c9d52f1f2c77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7d7509-2mjmq\" (UID: \"4b564ab7-13d0-4543-ab0a-c9d52f1f2c77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d7509-2mjmq" Sep 11 18:34:53.319042 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:53.319003 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d7509-2mjmq" Sep 11 18:34:53.596369 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:53.596338 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d7509-2mjmq"] Sep 11 18:34:53.597834 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:34:53.597802 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b564ab7_13d0_4543_ab0a_c9d52f1f2c77.slice/crio-59d5077ae534e164988a42540001429b34d5286757b750977d79f7e78d8106f5 WatchSource:0}: Error finding container 59d5077ae534e164988a42540001429b34d5286757b750977d79f7e78d8106f5: Status 404 returned error can't find the container with id 59d5077ae534e164988a42540001429b34d5286757b750977d79f7e78d8106f5 Sep 11 18:34:54.504487 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:54.504441 2561 generic.go:356] "Generic (PLEG): container finished" podID="4b564ab7-13d0-4543-ab0a-c9d52f1f2c77" containerID="b0b7b41054e8e6da8b22209803065ee1528600c04011c9b30721a9a979b16713" exitCode=0 Sep 11 18:34:54.505003 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:54.504535 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d7509-2mjmq" event={"ID":"4b564ab7-13d0-4543-ab0a-c9d52f1f2c77","Type":"ContainerDied","Data":"b0b7b41054e8e6da8b22209803065ee1528600c04011c9b30721a9a979b16713"} Sep 11 18:34:54.505003 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:54.504589 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d7509-2mjmq" event={"ID":"4b564ab7-13d0-4543-ab0a-c9d52f1f2c77","Type":"ContainerStarted","Data":"59d5077ae534e164988a42540001429b34d5286757b750977d79f7e78d8106f5"} Sep 11 18:34:55.660983 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:55.660953 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d7509-2mjmq" Sep 11 18:34:55.697180 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:55.697150 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b564ab7-13d0-4543-ab0a-c9d52f1f2c77-maas-api-service-ca\") pod \"4b564ab7-13d0-4543-ab0a-c9d52f1f2c77\" (UID: \"4b564ab7-13d0-4543-ab0a-c9d52f1f2c77\") " Sep 11 18:34:55.697516 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:55.697489 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b564ab7-13d0-4543-ab0a-c9d52f1f2c77-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4b564ab7-13d0-4543-ab0a-c9d52f1f2c77" (UID: "4b564ab7-13d0-4543-ab0a-c9d52f1f2c77"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:34:55.798138 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:55.798046 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b564ab7-13d0-4543-ab0a-c9d52f1f2c77-maas-api-service-ca\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:34:56.519756 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:56.519721 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d7509-2mjmq" Sep 11 18:34:56.519756 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:56.519746 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d7509-2mjmq" event={"ID":"4b564ab7-13d0-4543-ab0a-c9d52f1f2c77","Type":"ContainerDied","Data":"59d5077ae534e164988a42540001429b34d5286757b750977d79f7e78d8106f5"} Sep 11 18:34:56.519971 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:34:56.519800 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59d5077ae534e164988a42540001429b34d5286757b750977d79f7e78d8106f5" Sep 11 18:35:00.252911 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:00.252876 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-1de19d/test-model-1de19d-kserve-8774f9845-58nlj"] Sep 11 18:35:03.806691 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:03.806650 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d7509-2mjmq"] Sep 11 18:35:03.809621 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:03.809593 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d7509-2mjmq"] Sep 11 18:35:05.141117 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:05.141086 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b564ab7-13d0-4543-ab0a-c9d52f1f2c77" path="/var/lib/kubelet/pods/4b564ab7-13d0-4543-ab0a-c9d52f1f2c77/volumes" Sep 11 18:35:11.004556 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:11.004522 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s"] Sep 11 18:35:11.005014 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:11.004747 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s" podUID="11283c37-ec93-4ff1-a82b-ecc812c41b3d" containerName="istio-proxy" containerID="cri-o://ad1fbccb1aa2e54ac0091f14356d02bba2fd576f084bb858447f17deb4ead673" gracePeriod=30 Sep 11 18:35:11.348440 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:11.348358 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1de19d-mnghz"] Sep 11 18:35:11.348853 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:11.348839 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b564ab7-13d0-4543-ab0a-c9d52f1f2c77" containerName="revoke-keys" Sep 11 18:35:11.348901 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:11.348854 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b564ab7-13d0-4543-ab0a-c9d52f1f2c77" containerName="revoke-keys" Sep 11 18:35:11.348947 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:11.348938 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b564ab7-13d0-4543-ab0a-c9d52f1f2c77" containerName="revoke-keys" Sep 11 18:35:11.351393 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:11.351377 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1de19d-mnghz" Sep 11 18:35:11.353722 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:11.353700 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 18:35:11.353843 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:11.353702 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n968b\"" Sep 11 18:35:11.359683 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:11.359651 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1de19d-mnghz"] Sep 11 18:35:11.463908 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:11.463844 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14d99ecc-10d5-4932-a670-9b2bce996631-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1de19d-mnghz\" (UID: \"14d99ecc-10d5-4932-a670-9b2bce996631\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1de19d-mnghz" Sep 11 18:35:11.565426 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:11.565376 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14d99ecc-10d5-4932-a670-9b2bce996631-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1de19d-mnghz\" (UID: \"14d99ecc-10d5-4932-a670-9b2bce996631\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1de19d-mnghz" Sep 11 18:35:11.566039 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:11.566017 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14d99ecc-10d5-4932-a670-9b2bce996631-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1de19d-mnghz\" (UID: \"14d99ecc-10d5-4932-a670-9b2bce996631\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1de19d-mnghz" Sep 11 18:35:11.665899 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:11.665860 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1de19d-mnghz" Sep 11 18:35:11.944096 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:11.944068 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1de19d-mnghz"] Sep 11 18:35:11.945317 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:35:11.945274 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14d99ecc_10d5_4932_a670_9b2bce996631.slice/crio-b7cc0521c7485de63e8fd8c31b1e6f783bb17b59504d003ccd577b7fbcab8e66 WatchSource:0}: Error finding container b7cc0521c7485de63e8fd8c31b1e6f783bb17b59504d003ccd577b7fbcab8e66: Status 404 returned error can't find the container with id b7cc0521c7485de63e8fd8c31b1e6f783bb17b59504d003ccd577b7fbcab8e66 Sep 11 18:35:12.639630 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:12.639536 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1de19d-mnghz" event={"ID":"14d99ecc-10d5-4932-a670-9b2bce996631","Type":"ContainerStarted","Data":"22bc778d232453f5254340f465a641fc823e8a5ddf5f448b7344072dd84c91e5"} Sep 11 18:35:12.639630 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:12.639607 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1de19d-mnghz" event={"ID":"14d99ecc-10d5-4932-a670-9b2bce996631","Type":"ContainerStarted","Data":"b7cc0521c7485de63e8fd8c31b1e6f783bb17b59504d003ccd577b7fbcab8e66"} Sep 11 18:35:12.657099 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:12.657033 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1de19d-mnghz" podStartSLOduration=1.6570167580000001 podStartE2EDuration="1.657016758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:35:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:35:12.654429692 +0000 UTC m=+1194.151447448" watchObservedRunningTime="2026-09-11 18:35:12.657016758 +0000 UTC m=+1194.154034511" Sep 11 18:35:13.649222 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:13.649189 2561 generic.go:356] "Generic (PLEG): container finished" podID="14d99ecc-10d5-4932-a670-9b2bce996631" containerID="22bc778d232453f5254340f465a641fc823e8a5ddf5f448b7344072dd84c91e5" exitCode=0 Sep 11 18:35:13.649616 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:13.649270 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1de19d-mnghz" event={"ID":"14d99ecc-10d5-4932-a670-9b2bce996631","Type":"ContainerDied","Data":"22bc778d232453f5254340f465a641fc823e8a5ddf5f448b7344072dd84c91e5"} Sep 11 18:35:14.806411 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:14.806387 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1de19d-mnghz" Sep 11 18:35:14.902799 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:14.902767 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14d99ecc-10d5-4932-a670-9b2bce996631-maas-api-service-ca\") pod \"14d99ecc-10d5-4932-a670-9b2bce996631\" (UID: \"14d99ecc-10d5-4932-a670-9b2bce996631\") " Sep 11 18:35:14.903180 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:14.903155 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14d99ecc-10d5-4932-a670-9b2bce996631-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "14d99ecc-10d5-4932-a670-9b2bce996631" (UID: "14d99ecc-10d5-4932-a670-9b2bce996631"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:35:15.004219 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:15.004121 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14d99ecc-10d5-4932-a670-9b2bce996631-maas-api-service-ca\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:35:15.273120 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:15.273010 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-1de19d/test-model-1de19d-kserve-8774f9845-58nlj" podUID="e69c20c3-b07e-435d-adcd-1503fa87f3ce" containerName="main" containerID="cri-o://ca51057024ab02d24dedd9d65f821d660e39931bee3d8a0699d0d71372437380" gracePeriod=45 Sep 11 18:35:15.535257 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:15.535187 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1de19d/test-model-1de19d-kserve-8774f9845-58nlj" Sep 11 18:35:15.610870 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:15.610839 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e69c20c3-b07e-435d-adcd-1503fa87f3ce-tls-certs\") pod \"e69c20c3-b07e-435d-adcd-1503fa87f3ce\" (UID: \"e69c20c3-b07e-435d-adcd-1503fa87f3ce\") " Sep 11 18:35:15.611043 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:15.610885 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e69c20c3-b07e-435d-adcd-1503fa87f3ce-tmp-dir\") pod \"e69c20c3-b07e-435d-adcd-1503fa87f3ce\" (UID: \"e69c20c3-b07e-435d-adcd-1503fa87f3ce\") " Sep 11 18:35:15.611043 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:15.610924 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e69c20c3-b07e-435d-adcd-1503fa87f3ce-model-cache\") pod \"e69c20c3-b07e-435d-adcd-1503fa87f3ce\" (UID: \"e69c20c3-b07e-435d-adcd-1503fa87f3ce\") " Sep 11 18:35:15.611043 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:15.611017 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e69c20c3-b07e-435d-adcd-1503fa87f3ce-dshm\") pod \"e69c20c3-b07e-435d-adcd-1503fa87f3ce\" (UID: \"e69c20c3-b07e-435d-adcd-1503fa87f3ce\") " Sep 11 18:35:15.611253 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:15.611094 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mbg4d\" (UniqueName: \"kubernetes.io/projected/e69c20c3-b07e-435d-adcd-1503fa87f3ce-kube-api-access-mbg4d\") pod \"e69c20c3-b07e-435d-adcd-1503fa87f3ce\" (UID: \"e69c20c3-b07e-435d-adcd-1503fa87f3ce\") " Sep 11 18:35:15.611253 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:15.611149 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e69c20c3-b07e-435d-adcd-1503fa87f3ce-home\") pod \"e69c20c3-b07e-435d-adcd-1503fa87f3ce\" (UID: \"e69c20c3-b07e-435d-adcd-1503fa87f3ce\") " Sep 11 18:35:15.611253 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:15.611231 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e69c20c3-b07e-435d-adcd-1503fa87f3ce-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e69c20c3-b07e-435d-adcd-1503fa87f3ce" (UID: "e69c20c3-b07e-435d-adcd-1503fa87f3ce"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:35:15.611460 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:15.611430 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e69c20c3-b07e-435d-adcd-1503fa87f3ce-home" (OuterVolumeSpecName: "home") pod "e69c20c3-b07e-435d-adcd-1503fa87f3ce" (UID: "e69c20c3-b07e-435d-adcd-1503fa87f3ce"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:35:15.611573 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:15.611495 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e69c20c3-b07e-435d-adcd-1503fa87f3ce-model-cache" (OuterVolumeSpecName: "model-cache") pod "e69c20c3-b07e-435d-adcd-1503fa87f3ce" (UID: "e69c20c3-b07e-435d-adcd-1503fa87f3ce"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:35:15.611623 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:15.611598 2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e69c20c3-b07e-435d-adcd-1503fa87f3ce-home\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:35:15.611623 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:15.611609 2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e69c20c3-b07e-435d-adcd-1503fa87f3ce-tmp-dir\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:35:15.611623 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:15.611619 2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e69c20c3-b07e-435d-adcd-1503fa87f3ce-model-cache\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:35:15.613399 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:15.613371 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e69c20c3-b07e-435d-adcd-1503fa87f3ce-kube-api-access-mbg4d" (OuterVolumeSpecName: "kube-api-access-mbg4d") pod "e69c20c3-b07e-435d-adcd-1503fa87f3ce" (UID: "e69c20c3-b07e-435d-adcd-1503fa87f3ce"). InnerVolumeSpecName "kube-api-access-mbg4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:35:15.613506 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:15.613490 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e69c20c3-b07e-435d-adcd-1503fa87f3ce-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e69c20c3-b07e-435d-adcd-1503fa87f3ce" (UID: "e69c20c3-b07e-435d-adcd-1503fa87f3ce"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:35:15.613506 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:15.613494 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e69c20c3-b07e-435d-adcd-1503fa87f3ce-dshm" (OuterVolumeSpecName: "dshm") pod "e69c20c3-b07e-435d-adcd-1503fa87f3ce" (UID: "e69c20c3-b07e-435d-adcd-1503fa87f3ce"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:35:15.664491 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:15.664451 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1de19d-mnghz" Sep 11 18:35:15.664677 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:15.664490 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1de19d-mnghz" event={"ID":"14d99ecc-10d5-4932-a670-9b2bce996631","Type":"ContainerDied","Data":"b7cc0521c7485de63e8fd8c31b1e6f783bb17b59504d003ccd577b7fbcab8e66"} Sep 11 18:35:15.664677 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:15.664539 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7cc0521c7485de63e8fd8c31b1e6f783bb17b59504d003ccd577b7fbcab8e66" Sep 11 18:35:15.666106 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:15.666081 2561 generic.go:356] "Generic (PLEG): container finished" podID="e69c20c3-b07e-435d-adcd-1503fa87f3ce" containerID="ca51057024ab02d24dedd9d65f821d660e39931bee3d8a0699d0d71372437380" exitCode=0 Sep 11 18:35:15.666230 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:15.666111 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1de19d/test-model-1de19d-kserve-8774f9845-58nlj" Sep 11 18:35:15.666230 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:15.666149 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1de19d/test-model-1de19d-kserve-8774f9845-58nlj" event={"ID":"e69c20c3-b07e-435d-adcd-1503fa87f3ce","Type":"ContainerDied","Data":"ca51057024ab02d24dedd9d65f821d660e39931bee3d8a0699d0d71372437380"} Sep 11 18:35:15.666230 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:15.666188 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1de19d/test-model-1de19d-kserve-8774f9845-58nlj" event={"ID":"e69c20c3-b07e-435d-adcd-1503fa87f3ce","Type":"ContainerDied","Data":"2d04f348325780752db5f480a098a1003ca572df9ae0a8b0f26c0036eb0ac6b3"} Sep 11 18:35:15.666230 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:15.666214 2561 scope.go:117] "RemoveContainer" containerID="ca51057024ab02d24dedd9d65f821d660e39931bee3d8a0699d0d71372437380" Sep 11 18:35:15.676936 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:15.676918 2561 scope.go:117] "RemoveContainer" containerID="ca51057024ab02d24dedd9d65f821d660e39931bee3d8a0699d0d71372437380" Sep 11 18:35:15.677215 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:35:15.677197 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca51057024ab02d24dedd9d65f821d660e39931bee3d8a0699d0d71372437380\": container with ID starting with ca51057024ab02d24dedd9d65f821d660e39931bee3d8a0699d0d71372437380 not found: ID does not exist" containerID="ca51057024ab02d24dedd9d65f821d660e39931bee3d8a0699d0d71372437380" Sep 11 18:35:15.677278 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:15.677224 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca51057024ab02d24dedd9d65f821d660e39931bee3d8a0699d0d71372437380"} err="failed to get container status \"ca51057024ab02d24dedd9d65f821d660e39931bee3d8a0699d0d71372437380\": rpc error: code = NotFound desc = could not find container \"ca51057024ab02d24dedd9d65f821d660e39931bee3d8a0699d0d71372437380\": container with ID starting with ca51057024ab02d24dedd9d65f821d660e39931bee3d8a0699d0d71372437380 not found: ID does not exist" Sep 11 18:35:15.693082 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:15.693041 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-1de19d/test-model-1de19d-kserve-8774f9845-58nlj"] Sep 11 18:35:15.701672 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:15.701647 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-1de19d/test-model-1de19d-kserve-8774f9845-58nlj"] Sep 11 18:35:15.720997 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:15.720965 2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e69c20c3-b07e-435d-adcd-1503fa87f3ce-tls-certs\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:35:15.720997 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:15.720992 2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e69c20c3-b07e-435d-adcd-1503fa87f3ce-dshm\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:35:15.720997 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:15.721002 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbg4d\" (UniqueName: \"kubernetes.io/projected/e69c20c3-b07e-435d-adcd-1503fa87f3ce-kube-api-access-mbg4d\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:35:16.265261 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:16.265229 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s" Sep 11 18:35:16.429379 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:16.429343 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11283c37-ec93-4ff1-a82b-ecc812c41b3d-workload-certs\") pod \"11283c37-ec93-4ff1-a82b-ecc812c41b3d\" (UID: \"11283c37-ec93-4ff1-a82b-ecc812c41b3d\") " Sep 11 18:35:16.429573 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:16.429415 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11283c37-ec93-4ff1-a82b-ecc812c41b3d-istiod-ca-cert\") pod \"11283c37-ec93-4ff1-a82b-ecc812c41b3d\" (UID: \"11283c37-ec93-4ff1-a82b-ecc812c41b3d\") " Sep 11 18:35:16.429573 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:16.429445 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11283c37-ec93-4ff1-a82b-ecc812c41b3d-istio-podinfo\") pod \"11283c37-ec93-4ff1-a82b-ecc812c41b3d\" (UID: \"11283c37-ec93-4ff1-a82b-ecc812c41b3d\") " Sep 11 18:35:16.429573 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:16.429499 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11283c37-ec93-4ff1-a82b-ecc812c41b3d-workload-socket\") pod \"11283c37-ec93-4ff1-a82b-ecc812c41b3d\" (UID: \"11283c37-ec93-4ff1-a82b-ecc812c41b3d\") " Sep 11 18:35:16.429573 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:16.429533 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11283c37-ec93-4ff1-a82b-ecc812c41b3d-istio-token\") pod \"11283c37-ec93-4ff1-a82b-ecc812c41b3d\" (UID: \"11283c37-ec93-4ff1-a82b-ecc812c41b3d\") " Sep 11 18:35:16.429573 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:16.429568 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11283c37-ec93-4ff1-a82b-ecc812c41b3d-istio-data\") pod \"11283c37-ec93-4ff1-a82b-ecc812c41b3d\" (UID: \"11283c37-ec93-4ff1-a82b-ecc812c41b3d\") " Sep 11 18:35:16.429806 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:16.429646 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11283c37-ec93-4ff1-a82b-ecc812c41b3d-istio-envoy\") pod \"11283c37-ec93-4ff1-a82b-ecc812c41b3d\" (UID: \"11283c37-ec93-4ff1-a82b-ecc812c41b3d\") " Sep 11 18:35:16.429806 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:16.429671 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11283c37-ec93-4ff1-a82b-ecc812c41b3d-credential-socket\") pod \"11283c37-ec93-4ff1-a82b-ecc812c41b3d\" (UID: \"11283c37-ec93-4ff1-a82b-ecc812c41b3d\") " Sep 11 18:35:16.429806 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:16.429664 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11283c37-ec93-4ff1-a82b-ecc812c41b3d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "11283c37-ec93-4ff1-a82b-ecc812c41b3d" (UID: "11283c37-ec93-4ff1-a82b-ecc812c41b3d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:35:16.429806 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:16.429752 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11283c37-ec93-4ff1-a82b-ecc812c41b3d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "11283c37-ec93-4ff1-a82b-ecc812c41b3d" (UID: "11283c37-ec93-4ff1-a82b-ecc812c41b3d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:35:16.430036 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:16.429847 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11283c37-ec93-4ff1-a82b-ecc812c41b3d-istio-data" (OuterVolumeSpecName: "istio-data") pod "11283c37-ec93-4ff1-a82b-ecc812c41b3d" (UID: "11283c37-ec93-4ff1-a82b-ecc812c41b3d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:35:16.430036 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:16.429889 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11283c37-ec93-4ff1-a82b-ecc812c41b3d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "11283c37-ec93-4ff1-a82b-ecc812c41b3d" (UID: "11283c37-ec93-4ff1-a82b-ecc812c41b3d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:35:16.430036 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:16.429927 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tsdth\" (UniqueName: \"kubernetes.io/projected/11283c37-ec93-4ff1-a82b-ecc812c41b3d-kube-api-access-tsdth\") pod \"11283c37-ec93-4ff1-a82b-ecc812c41b3d\" (UID: \"11283c37-ec93-4ff1-a82b-ecc812c41b3d\") " Sep 11 18:35:16.430036 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:16.429979 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11283c37-ec93-4ff1-a82b-ecc812c41b3d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "11283c37-ec93-4ff1-a82b-ecc812c41b3d" (UID: "11283c37-ec93-4ff1-a82b-ecc812c41b3d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:35:16.430432 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:16.430410 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11283c37-ec93-4ff1-a82b-ecc812c41b3d-istiod-ca-cert\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:35:16.430523 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:16.430437 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11283c37-ec93-4ff1-a82b-ecc812c41b3d-workload-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:35:16.430523 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:16.430451 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11283c37-ec93-4ff1-a82b-ecc812c41b3d-istio-data\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:35:16.430523 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:16.430465 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11283c37-ec93-4ff1-a82b-ecc812c41b3d-credential-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:35:16.430523 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:16.430479 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11283c37-ec93-4ff1-a82b-ecc812c41b3d-workload-certs\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:35:16.431750 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:16.431719 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11283c37-ec93-4ff1-a82b-ecc812c41b3d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "11283c37-ec93-4ff1-a82b-ecc812c41b3d" (UID: "11283c37-ec93-4ff1-a82b-ecc812c41b3d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:35:16.431750 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:16.431743 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11283c37-ec93-4ff1-a82b-ecc812c41b3d-istio-token" (OuterVolumeSpecName: "istio-token") pod "11283c37-ec93-4ff1-a82b-ecc812c41b3d" (UID: "11283c37-ec93-4ff1-a82b-ecc812c41b3d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:35:16.431906 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:16.431781 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11283c37-ec93-4ff1-a82b-ecc812c41b3d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "11283c37-ec93-4ff1-a82b-ecc812c41b3d" (UID: "11283c37-ec93-4ff1-a82b-ecc812c41b3d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 18:35:16.431906 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:16.431893 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11283c37-ec93-4ff1-a82b-ecc812c41b3d-kube-api-access-tsdth" (OuterVolumeSpecName: "kube-api-access-tsdth") pod "11283c37-ec93-4ff1-a82b-ecc812c41b3d" (UID: "11283c37-ec93-4ff1-a82b-ecc812c41b3d"). InnerVolumeSpecName "kube-api-access-tsdth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:35:16.531903 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:16.531863 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11283c37-ec93-4ff1-a82b-ecc812c41b3d-istio-token\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:35:16.531903 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:16.531898 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11283c37-ec93-4ff1-a82b-ecc812c41b3d-istio-envoy\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:35:16.531903 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:16.531908 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsdth\" (UniqueName: \"kubernetes.io/projected/11283c37-ec93-4ff1-a82b-ecc812c41b3d-kube-api-access-tsdth\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:35:16.532187 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:16.531918 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11283c37-ec93-4ff1-a82b-ecc812c41b3d-istio-podinfo\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:35:16.675012 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:16.674979 2561 generic.go:356] "Generic (PLEG): container finished" podID="11283c37-ec93-4ff1-a82b-ecc812c41b3d" containerID="ad1fbccb1aa2e54ac0091f14356d02bba2fd576f084bb858447f17deb4ead673" exitCode=0 Sep 11 18:35:16.675246 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:16.675047 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s" Sep 11 18:35:16.675246 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:16.675082 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s" event={"ID":"11283c37-ec93-4ff1-a82b-ecc812c41b3d","Type":"ContainerDied","Data":"ad1fbccb1aa2e54ac0091f14356d02bba2fd576f084bb858447f17deb4ead673"} Sep 11 18:35:16.675246 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:16.675193 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s" event={"ID":"11283c37-ec93-4ff1-a82b-ecc812c41b3d","Type":"ContainerDied","Data":"6f63f3e97743ec53fab855d0965b87c649982cee4e9ba8ea008b13bc13107ecc"} Sep 11 18:35:16.675246 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:16.675232 2561 scope.go:117] "RemoveContainer" containerID="ad1fbccb1aa2e54ac0091f14356d02bba2fd576f084bb858447f17deb4ead673" Sep 11 18:35:16.686508 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:16.686481 2561 scope.go:117] "RemoveContainer" containerID="4d7d1845627fb9b2afcf35ce69e88bae290d6c42957bb9c7184ae3f53c2bf247" Sep 11 18:35:16.696018 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:16.695995 2561 scope.go:117] "RemoveContainer" containerID="ad1fbccb1aa2e54ac0091f14356d02bba2fd576f084bb858447f17deb4ead673" Sep 11 18:35:16.696327 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:35:16.696307 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad1fbccb1aa2e54ac0091f14356d02bba2fd576f084bb858447f17deb4ead673\": container with ID starting with ad1fbccb1aa2e54ac0091f14356d02bba2fd576f084bb858447f17deb4ead673 not found: ID does not exist" containerID="ad1fbccb1aa2e54ac0091f14356d02bba2fd576f084bb858447f17deb4ead673" Sep 11 18:35:16.696385 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:16.696338 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad1fbccb1aa2e54ac0091f14356d02bba2fd576f084bb858447f17deb4ead673"} err="failed to get container status \"ad1fbccb1aa2e54ac0091f14356d02bba2fd576f084bb858447f17deb4ead673\": rpc error: code = NotFound desc = could not find container \"ad1fbccb1aa2e54ac0091f14356d02bba2fd576f084bb858447f17deb4ead673\": container with ID starting with ad1fbccb1aa2e54ac0091f14356d02bba2fd576f084bb858447f17deb4ead673 not found: ID does not exist" Sep 11 18:35:16.696385 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:16.696357 2561 scope.go:117] "RemoveContainer" containerID="4d7d1845627fb9b2afcf35ce69e88bae290d6c42957bb9c7184ae3f53c2bf247" Sep 11 18:35:16.696709 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:35:16.696684 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d7d1845627fb9b2afcf35ce69e88bae290d6c42957bb9c7184ae3f53c2bf247\": container with ID starting with 4d7d1845627fb9b2afcf35ce69e88bae290d6c42957bb9c7184ae3f53c2bf247 not found: ID does not exist" containerID="4d7d1845627fb9b2afcf35ce69e88bae290d6c42957bb9c7184ae3f53c2bf247" Sep 11 18:35:16.696836 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:16.696717 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d7d1845627fb9b2afcf35ce69e88bae290d6c42957bb9c7184ae3f53c2bf247"} err="failed to get container status \"4d7d1845627fb9b2afcf35ce69e88bae290d6c42957bb9c7184ae3f53c2bf247\": rpc error: code = NotFound desc = could not find container \"4d7d1845627fb9b2afcf35ce69e88bae290d6c42957bb9c7184ae3f53c2bf247\": container with ID starting with 4d7d1845627fb9b2afcf35ce69e88bae290d6c42957bb9c7184ae3f53c2bf247 not found: ID does not exist" Sep 11 18:35:16.700648 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:16.700623 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s"] Sep 11 18:35:16.709637 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:16.709614 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-7d7509-openshift-default-67479fd5d-z6r6s"] Sep 11 18:35:17.140080 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:17.139976 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11283c37-ec93-4ff1-a82b-ecc812c41b3d" path="/var/lib/kubelet/pods/11283c37-ec93-4ff1-a82b-ecc812c41b3d/volumes" Sep 11 18:35:17.140654 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:17.140568 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e69c20c3-b07e-435d-adcd-1503fa87f3ce" path="/var/lib/kubelet/pods/e69c20c3-b07e-435d-adcd-1503fa87f3ce/volumes" Sep 11 18:35:19.433599 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:19.433568 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dj524_60702ef3-98ef-463b-b877-3a00825353d8/console-operator/1.log" Sep 11 18:35:19.438358 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:19.438334 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dj524_60702ef3-98ef-463b-b877-3a00825353d8/console-operator/1.log" Sep 11 18:35:19.445632 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:19.445607 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4knd_42e5ddd8-a96c-45a1-adf9-caa47ee902db/ovn-acl-logging/0.log" Sep 11 18:35:19.450426 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:19.450405 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4knd_42e5ddd8-a96c-45a1-adf9-caa47ee902db/ovn-acl-logging/0.log" Sep 11 18:35:19.515897 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:19.515865 2561 scope.go:117] "RemoveContainer" containerID="c4dd24bbb5d36e1ecc1867c6fb2ba46ac67ede62b254a18eb6cb3086928ab0e8" Sep 11 18:35:21.932589 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:21.932550 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1de19d-8659676f7d-q4qt4"] Sep 11 18:35:21.933093 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:21.932801 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1de19d-8659676f7d-q4qt4" podUID="68ef7a27-900e-4d80-8c96-023cb2e14f21" containerName="payload-pre-processing" containerID="cri-o://586bedfb21cc9e01eea07997973bb96589519a47403fb6ed8d20420e5d22d8f3" gracePeriod=30 Sep 11 18:35:22.110923 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:22.110895 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1de19d-mnghz"] Sep 11 18:35:22.116086 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:22.116033 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1de19d-mnghz"] Sep 11 18:35:22.206035 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:22.206006 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1de19d-8659676f7d-q4qt4" Sep 11 18:35:22.296922 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:22.296887 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68ef7a27-900e-4d80-8c96-023cb2e14f21-plugins-config-volume\") pod \"68ef7a27-900e-4d80-8c96-023cb2e14f21\" (UID: \"68ef7a27-900e-4d80-8c96-023cb2e14f21\") " Sep 11 18:35:22.297121 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:22.297013 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lk56b\" (UniqueName: \"kubernetes.io/projected/68ef7a27-900e-4d80-8c96-023cb2e14f21-kube-api-access-lk56b\") pod \"68ef7a27-900e-4d80-8c96-023cb2e14f21\" (UID: \"68ef7a27-900e-4d80-8c96-023cb2e14f21\") " Sep 11 18:35:22.297337 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:22.297291 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68ef7a27-900e-4d80-8c96-023cb2e14f21-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "68ef7a27-900e-4d80-8c96-023cb2e14f21" (UID: "68ef7a27-900e-4d80-8c96-023cb2e14f21"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:35:22.299158 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:22.299130 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68ef7a27-900e-4d80-8c96-023cb2e14f21-kube-api-access-lk56b" (OuterVolumeSpecName: "kube-api-access-lk56b") pod "68ef7a27-900e-4d80-8c96-023cb2e14f21" (UID: "68ef7a27-900e-4d80-8c96-023cb2e14f21"). InnerVolumeSpecName "kube-api-access-lk56b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:35:22.398642 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:22.398601 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk56b\" (UniqueName: \"kubernetes.io/projected/68ef7a27-900e-4d80-8c96-023cb2e14f21-kube-api-access-lk56b\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:35:22.398642 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:22.398636 2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68ef7a27-900e-4d80-8c96-023cb2e14f21-plugins-config-volume\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:35:22.723672 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:22.723634 2561 generic.go:356] "Generic (PLEG): container finished" podID="68ef7a27-900e-4d80-8c96-023cb2e14f21" containerID="586bedfb21cc9e01eea07997973bb96589519a47403fb6ed8d20420e5d22d8f3" exitCode=0 Sep 11 18:35:22.723884 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:22.723679 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1de19d-8659676f7d-q4qt4" Sep 11 18:35:22.723884 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:22.723711 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1de19d-8659676f7d-q4qt4" event={"ID":"68ef7a27-900e-4d80-8c96-023cb2e14f21","Type":"ContainerDied","Data":"586bedfb21cc9e01eea07997973bb96589519a47403fb6ed8d20420e5d22d8f3"} Sep 11 18:35:22.723884 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:22.723768 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1de19d-8659676f7d-q4qt4" event={"ID":"68ef7a27-900e-4d80-8c96-023cb2e14f21","Type":"ContainerDied","Data":"1ef353126260cb2f598d0ab247f4f4b0553939c3573aa99fbaef19fc5074faf0"} Sep 11 18:35:22.723884 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:22.723799 2561 scope.go:117] "RemoveContainer" containerID="586bedfb21cc9e01eea07997973bb96589519a47403fb6ed8d20420e5d22d8f3" Sep 11 18:35:22.734281 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:22.734260 2561 scope.go:117] "RemoveContainer" containerID="586bedfb21cc9e01eea07997973bb96589519a47403fb6ed8d20420e5d22d8f3" Sep 11 18:35:22.734583 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:35:22.734564 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"586bedfb21cc9e01eea07997973bb96589519a47403fb6ed8d20420e5d22d8f3\": container with ID starting with 586bedfb21cc9e01eea07997973bb96589519a47403fb6ed8d20420e5d22d8f3 not found: ID does not exist" containerID="586bedfb21cc9e01eea07997973bb96589519a47403fb6ed8d20420e5d22d8f3" Sep 11 18:35:22.734635 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:22.734592 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"586bedfb21cc9e01eea07997973bb96589519a47403fb6ed8d20420e5d22d8f3"} err="failed to get container status \"586bedfb21cc9e01eea07997973bb96589519a47403fb6ed8d20420e5d22d8f3\": rpc error: code = NotFound desc = could not find container \"586bedfb21cc9e01eea07997973bb96589519a47403fb6ed8d20420e5d22d8f3\": container with ID starting with 586bedfb21cc9e01eea07997973bb96589519a47403fb6ed8d20420e5d22d8f3 not found: ID does not exist" Sep 11 18:35:22.749670 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:22.749636 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1de19d-8659676f7d-q4qt4"] Sep 11 18:35:22.754380 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:22.754348 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1de19d-8659676f7d-q4qt4"] Sep 11 18:35:23.140850 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:23.140765 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14d99ecc-10d5-4932-a670-9b2bce996631" path="/var/lib/kubelet/pods/14d99ecc-10d5-4932-a670-9b2bce996631/volumes" Sep 11 18:35:23.141272 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:23.141258 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68ef7a27-900e-4d80-8c96-023cb2e14f21" path="/var/lib/kubelet/pods/68ef7a27-900e-4d80-8c96-023cb2e14f21/volumes" Sep 11 18:35:28.813008 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:28.812969 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd"] Sep 11 18:35:28.813455 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:28.813220 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd" podUID="7e9b9a66-30ad-4d94-95aa-d09ee8eea072" containerName="istio-proxy" containerID="cri-o://041314b9ddd7eb2d28344155844f12a64cd53693deb0b1a532717b2252a1b598" gracePeriod=30 Sep 11 18:35:34.078078 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:34.077993 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd" Sep 11 18:35:34.124980 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:34.124944 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k5kg5\" (UniqueName: \"kubernetes.io/projected/7e9b9a66-30ad-4d94-95aa-d09ee8eea072-kube-api-access-k5kg5\") pod \"7e9b9a66-30ad-4d94-95aa-d09ee8eea072\" (UID: \"7e9b9a66-30ad-4d94-95aa-d09ee8eea072\") " Sep 11 18:35:34.125175 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:34.125033 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e9b9a66-30ad-4d94-95aa-d09ee8eea072-credential-socket\") pod \"7e9b9a66-30ad-4d94-95aa-d09ee8eea072\" (UID: \"7e9b9a66-30ad-4d94-95aa-d09ee8eea072\") " Sep 11 18:35:34.125175 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:34.125085 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e9b9a66-30ad-4d94-95aa-d09ee8eea072-istio-token\") pod \"7e9b9a66-30ad-4d94-95aa-d09ee8eea072\" (UID: \"7e9b9a66-30ad-4d94-95aa-d09ee8eea072\") " Sep 11 18:35:34.125175 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:34.125129 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e9b9a66-30ad-4d94-95aa-d09ee8eea072-workload-certs\") pod \"7e9b9a66-30ad-4d94-95aa-d09ee8eea072\" (UID: \"7e9b9a66-30ad-4d94-95aa-d09ee8eea072\") " Sep 11 18:35:34.125303 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:34.125272 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e9b9a66-30ad-4d94-95aa-d09ee8eea072-istio-data\") pod \"7e9b9a66-30ad-4d94-95aa-d09ee8eea072\" (UID: \"7e9b9a66-30ad-4d94-95aa-d09ee8eea072\") " Sep 11 18:35:34.125355 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:34.125340 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e9b9a66-30ad-4d94-95aa-d09ee8eea072-istio-podinfo\") pod \"7e9b9a66-30ad-4d94-95aa-d09ee8eea072\" (UID: \"7e9b9a66-30ad-4d94-95aa-d09ee8eea072\") " Sep 11 18:35:34.125397 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:34.125378 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e9b9a66-30ad-4d94-95aa-d09ee8eea072-istiod-ca-cert\") pod \"7e9b9a66-30ad-4d94-95aa-d09ee8eea072\" (UID: \"7e9b9a66-30ad-4d94-95aa-d09ee8eea072\") " Sep 11 18:35:34.125451 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:34.125392 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e9b9a66-30ad-4d94-95aa-d09ee8eea072-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7e9b9a66-30ad-4d94-95aa-d09ee8eea072" (UID: "7e9b9a66-30ad-4d94-95aa-d09ee8eea072"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:35:34.125451 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:34.125409 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e9b9a66-30ad-4d94-95aa-d09ee8eea072-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7e9b9a66-30ad-4d94-95aa-d09ee8eea072" (UID: "7e9b9a66-30ad-4d94-95aa-d09ee8eea072"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:35:34.125451 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:34.125446 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e9b9a66-30ad-4d94-95aa-d09ee8eea072-istio-envoy\") pod \"7e9b9a66-30ad-4d94-95aa-d09ee8eea072\" (UID: \"7e9b9a66-30ad-4d94-95aa-d09ee8eea072\") " Sep 11 18:35:34.125611 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:34.125512 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e9b9a66-30ad-4d94-95aa-d09ee8eea072-workload-socket\") pod \"7e9b9a66-30ad-4d94-95aa-d09ee8eea072\" (UID: \"7e9b9a66-30ad-4d94-95aa-d09ee8eea072\") " Sep 11 18:35:34.125611 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:34.125521 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e9b9a66-30ad-4d94-95aa-d09ee8eea072-istio-data" (OuterVolumeSpecName: "istio-data") pod "7e9b9a66-30ad-4d94-95aa-d09ee8eea072" (UID: "7e9b9a66-30ad-4d94-95aa-d09ee8eea072"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:35:34.125828 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:34.125799 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e9b9a66-30ad-4d94-95aa-d09ee8eea072-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7e9b9a66-30ad-4d94-95aa-d09ee8eea072" (UID: "7e9b9a66-30ad-4d94-95aa-d09ee8eea072"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:35:34.125958 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:34.125940 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e9b9a66-30ad-4d94-95aa-d09ee8eea072-workload-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:35:34.126030 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:34.125960 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e9b9a66-30ad-4d94-95aa-d09ee8eea072-credential-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:35:34.126030 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:34.125969 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e9b9a66-30ad-4d94-95aa-d09ee8eea072-workload-certs\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:35:34.126030 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:34.125977 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e9b9a66-30ad-4d94-95aa-d09ee8eea072-istio-data\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:35:34.126222 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:34.126028 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e9b9a66-30ad-4d94-95aa-d09ee8eea072-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7e9b9a66-30ad-4d94-95aa-d09ee8eea072" (UID: "7e9b9a66-30ad-4d94-95aa-d09ee8eea072"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:35:34.127320 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:34.127297 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e9b9a66-30ad-4d94-95aa-d09ee8eea072-kube-api-access-k5kg5" (OuterVolumeSpecName: "kube-api-access-k5kg5") pod "7e9b9a66-30ad-4d94-95aa-d09ee8eea072" (UID: "7e9b9a66-30ad-4d94-95aa-d09ee8eea072"). InnerVolumeSpecName "kube-api-access-k5kg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:35:34.127552 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:34.127531 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e9b9a66-30ad-4d94-95aa-d09ee8eea072-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7e9b9a66-30ad-4d94-95aa-d09ee8eea072" (UID: "7e9b9a66-30ad-4d94-95aa-d09ee8eea072"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 18:35:34.127675 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:34.127650 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e9b9a66-30ad-4d94-95aa-d09ee8eea072-istio-token" (OuterVolumeSpecName: "istio-token") pod "7e9b9a66-30ad-4d94-95aa-d09ee8eea072" (UID: "7e9b9a66-30ad-4d94-95aa-d09ee8eea072"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:35:34.127773 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:34.127689 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e9b9a66-30ad-4d94-95aa-d09ee8eea072-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7e9b9a66-30ad-4d94-95aa-d09ee8eea072" (UID: "7e9b9a66-30ad-4d94-95aa-d09ee8eea072"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:35:34.227585 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:34.227550 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e9b9a66-30ad-4d94-95aa-d09ee8eea072-istio-envoy\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:35:34.227585 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:34.227584 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5kg5\" (UniqueName: \"kubernetes.io/projected/7e9b9a66-30ad-4d94-95aa-d09ee8eea072-kube-api-access-k5kg5\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:35:34.227585 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:34.227601 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e9b9a66-30ad-4d94-95aa-d09ee8eea072-istio-token\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:35:34.227837 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:34.227615 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e9b9a66-30ad-4d94-95aa-d09ee8eea072-istio-podinfo\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:35:34.227837 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:34.227628 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e9b9a66-30ad-4d94-95aa-d09ee8eea072-istiod-ca-cert\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:35:34.836785 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:34.836753 2561 generic.go:356] "Generic (PLEG): container finished" podID="7e9b9a66-30ad-4d94-95aa-d09ee8eea072" containerID="041314b9ddd7eb2d28344155844f12a64cd53693deb0b1a532717b2252a1b598" exitCode=0 Sep 11 18:35:34.836977 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:34.836811 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd" Sep 11 18:35:34.836977 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:34.836844 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd" event={"ID":"7e9b9a66-30ad-4d94-95aa-d09ee8eea072","Type":"ContainerDied","Data":"041314b9ddd7eb2d28344155844f12a64cd53693deb0b1a532717b2252a1b598"} Sep 11 18:35:34.836977 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:34.836902 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd" event={"ID":"7e9b9a66-30ad-4d94-95aa-d09ee8eea072","Type":"ContainerDied","Data":"00813ffb68883db053e0e8c98fb49397c7f439c4e093747ad3b7c6fbac8fa3be"} Sep 11 18:35:34.836977 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:34.836926 2561 scope.go:117] "RemoveContainer" containerID="041314b9ddd7eb2d28344155844f12a64cd53693deb0b1a532717b2252a1b598" Sep 11 18:35:34.847892 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:34.847862 2561 scope.go:117] "RemoveContainer" containerID="d695c9409d060d75a4b2891eb4a3090da2853e05ec90a20216eaaa7c45046e8c" Sep 11 18:35:34.860262 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:34.860236 2561 scope.go:117] "RemoveContainer" containerID="041314b9ddd7eb2d28344155844f12a64cd53693deb0b1a532717b2252a1b598" Sep 11 18:35:34.860723 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:35:34.860701 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"041314b9ddd7eb2d28344155844f12a64cd53693deb0b1a532717b2252a1b598\": container with ID starting with 041314b9ddd7eb2d28344155844f12a64cd53693deb0b1a532717b2252a1b598 not found: ID does not exist" containerID="041314b9ddd7eb2d28344155844f12a64cd53693deb0b1a532717b2252a1b598" Sep 11 18:35:34.860723 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:34.860732 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"041314b9ddd7eb2d28344155844f12a64cd53693deb0b1a532717b2252a1b598"} err="failed to get container status \"041314b9ddd7eb2d28344155844f12a64cd53693deb0b1a532717b2252a1b598\": rpc error: code = NotFound desc = could not find container \"041314b9ddd7eb2d28344155844f12a64cd53693deb0b1a532717b2252a1b598\": container with ID starting with 041314b9ddd7eb2d28344155844f12a64cd53693deb0b1a532717b2252a1b598 not found: ID does not exist" Sep 11 18:35:34.860875 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:34.860753 2561 scope.go:117] "RemoveContainer" containerID="d695c9409d060d75a4b2891eb4a3090da2853e05ec90a20216eaaa7c45046e8c" Sep 11 18:35:34.861379 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:35:34.861356 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d695c9409d060d75a4b2891eb4a3090da2853e05ec90a20216eaaa7c45046e8c\": container with ID starting with d695c9409d060d75a4b2891eb4a3090da2853e05ec90a20216eaaa7c45046e8c not found: ID does not exist" containerID="d695c9409d060d75a4b2891eb4a3090da2853e05ec90a20216eaaa7c45046e8c" Sep 11 18:35:34.861441 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:34.861387 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d695c9409d060d75a4b2891eb4a3090da2853e05ec90a20216eaaa7c45046e8c"} err="failed to get container status \"d695c9409d060d75a4b2891eb4a3090da2853e05ec90a20216eaaa7c45046e8c\": rpc error: code = NotFound desc = could not find container \"d695c9409d060d75a4b2891eb4a3090da2853e05ec90a20216eaaa7c45046e8c\": container with ID starting with d695c9409d060d75a4b2891eb4a3090da2853e05ec90a20216eaaa7c45046e8c not found: ID does not exist" Sep 11 18:35:34.863493 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:34.863471 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd"] Sep 11 18:35:34.872306 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:34.872277 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-1de19d-openshift-default-56df49b4b4-99jgd"] Sep 11 18:35:35.140022 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:35.139935 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e9b9a66-30ad-4d94-95aa-d09ee8eea072" path="/var/lib/kubelet/pods/7e9b9a66-30ad-4d94-95aa-d09ee8eea072/volumes" Sep 11 18:35:37.329165 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:37.329124 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7ca6df-5c896694c5-qf4zs"] Sep 11 18:35:37.329631 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:37.329577 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e9b9a66-30ad-4d94-95aa-d09ee8eea072" containerName="istio-proxy" Sep 11 18:35:37.329631 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:37.329590 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e9b9a66-30ad-4d94-95aa-d09ee8eea072" containerName="istio-proxy" Sep 11 18:35:37.329631 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:37.329612 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68ef7a27-900e-4d80-8c96-023cb2e14f21" containerName="payload-pre-processing" Sep 11 18:35:37.329631 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:37.329618 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="68ef7a27-900e-4d80-8c96-023cb2e14f21" containerName="payload-pre-processing" Sep 11 18:35:37.329631 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:37.329627 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e69c20c3-b07e-435d-adcd-1503fa87f3ce" containerName="main" Sep 11 18:35:37.329631 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:37.329633 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e69c20c3-b07e-435d-adcd-1503fa87f3ce" containerName="main" Sep 11 18:35:37.329944 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:37.329642 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14d99ecc-10d5-4932-a670-9b2bce996631" containerName="revoke-keys" Sep 11 18:35:37.329944 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:37.329647 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="14d99ecc-10d5-4932-a670-9b2bce996631" containerName="revoke-keys" Sep 11 18:35:37.329944 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:37.329655 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11283c37-ec93-4ff1-a82b-ecc812c41b3d" containerName="istio-proxy" Sep 11 18:35:37.329944 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:37.329660 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="11283c37-ec93-4ff1-a82b-ecc812c41b3d" containerName="istio-proxy" Sep 11 18:35:37.329944 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:37.329670 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11283c37-ec93-4ff1-a82b-ecc812c41b3d" containerName="istio-proxy" Sep 11 18:35:37.329944 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:37.329676 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="11283c37-ec93-4ff1-a82b-ecc812c41b3d" containerName="istio-proxy" Sep 11 18:35:37.329944 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:37.329736 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="14d99ecc-10d5-4932-a670-9b2bce996631" containerName="revoke-keys" Sep 11 18:35:37.329944 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:37.329744 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e9b9a66-30ad-4d94-95aa-d09ee8eea072" containerName="istio-proxy" Sep 11 18:35:37.329944 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:37.329750 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="68ef7a27-900e-4d80-8c96-023cb2e14f21" containerName="payload-pre-processing" Sep 11 18:35:37.329944 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:37.329757 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e69c20c3-b07e-435d-adcd-1503fa87f3ce" containerName="main" Sep 11 18:35:37.329944 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:37.329763 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="11283c37-ec93-4ff1-a82b-ecc812c41b3d" containerName="istio-proxy" Sep 11 18:35:37.329944 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:37.329768 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="11283c37-ec93-4ff1-a82b-ecc812c41b3d" containerName="istio-proxy" Sep 11 18:35:37.332896 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:37.332870 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7ca6df-5c896694c5-qf4zs" Sep 11 18:35:37.335716 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:37.335399 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-7ca6df-serving-cert\"" Sep 11 18:35:37.335716 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:37.335501 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 11 18:35:37.335716 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:37.335516 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-c9xd9\"" Sep 11 18:35:37.335716 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:37.335570 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 18:35:37.335716 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:37.335630 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 11 18:35:37.343902 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:37.343877 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7ca6df-5c896694c5-qf4zs"] Sep 11 18:35:37.360287 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:37.360256 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b4dec75-b53b-4e05-82eb-553ac25bee7d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-7ca6df-5c896694c5-qf4zs\" (UID: \"0b4dec75-b53b-4e05-82eb-553ac25bee7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7ca6df-5c896694c5-qf4zs" Sep 11 18:35:37.360497 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:37.360312 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b4dec75-b53b-4e05-82eb-553ac25bee7d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-7ca6df-5c896694c5-qf4zs\" (UID: \"0b4dec75-b53b-4e05-82eb-553ac25bee7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7ca6df-5c896694c5-qf4zs" Sep 11 18:35:37.360497 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:37.360346 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n9gw\" (UniqueName: \"kubernetes.io/projected/0b4dec75-b53b-4e05-82eb-553ac25bee7d-kube-api-access-6n9gw\") pod \"maas-api-e2e-shared-a-w3-7ca6df-5c896694c5-qf4zs\" (UID: \"0b4dec75-b53b-4e05-82eb-553ac25bee7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7ca6df-5c896694c5-qf4zs" Sep 11 18:35:37.360497 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:37.360411 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b4dec75-b53b-4e05-82eb-553ac25bee7d-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-7ca6df-5c896694c5-qf4zs\" (UID: \"0b4dec75-b53b-4e05-82eb-553ac25bee7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7ca6df-5c896694c5-qf4zs" Sep 11 18:35:37.461108 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:37.461035 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b4dec75-b53b-4e05-82eb-553ac25bee7d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-7ca6df-5c896694c5-qf4zs\" (UID: \"0b4dec75-b53b-4e05-82eb-553ac25bee7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7ca6df-5c896694c5-qf4zs" Sep 11 18:35:37.461290 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:37.461119 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n9gw\" (UniqueName: \"kubernetes.io/projected/0b4dec75-b53b-4e05-82eb-553ac25bee7d-kube-api-access-6n9gw\") pod \"maas-api-e2e-shared-a-w3-7ca6df-5c896694c5-qf4zs\" (UID: \"0b4dec75-b53b-4e05-82eb-553ac25bee7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7ca6df-5c896694c5-qf4zs" Sep 11 18:35:37.461290 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:35:37.461192 2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7ca6df-serving-cert: secret "maas-api-e2e-shared-a-w3-7ca6df-serving-cert" not found Sep 11 18:35:37.461290 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:35:37.461258 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b4dec75-b53b-4e05-82eb-553ac25bee7d-maas-api-tls podName:0b4dec75-b53b-4e05-82eb-553ac25bee7d nodeName:}" failed. No retries permitted until 2026-09-11 18:35:37.961240166 +0000 UTC m=+1219.458257898 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0b4dec75-b53b-4e05-82eb-553ac25bee7d-maas-api-tls") pod "maas-api-e2e-shared-a-w3-7ca6df-5c896694c5-qf4zs" (UID: "0b4dec75-b53b-4e05-82eb-553ac25bee7d") : secret "maas-api-e2e-shared-a-w3-7ca6df-serving-cert" not found Sep 11 18:35:37.461290 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:37.461277 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b4dec75-b53b-4e05-82eb-553ac25bee7d-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-7ca6df-5c896694c5-qf4zs\" (UID: \"0b4dec75-b53b-4e05-82eb-553ac25bee7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7ca6df-5c896694c5-qf4zs" Sep 11 18:35:37.461493 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:37.461409 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b4dec75-b53b-4e05-82eb-553ac25bee7d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-7ca6df-5c896694c5-qf4zs\" (UID: \"0b4dec75-b53b-4e05-82eb-553ac25bee7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7ca6df-5c896694c5-qf4zs" Sep 11 18:35:37.462046 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:37.462025 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b4dec75-b53b-4e05-82eb-553ac25bee7d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-7ca6df-5c896694c5-qf4zs\" (UID: \"0b4dec75-b53b-4e05-82eb-553ac25bee7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7ca6df-5c896694c5-qf4zs" Sep 11 18:35:37.463873 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:37.463847 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b4dec75-b53b-4e05-82eb-553ac25bee7d-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-7ca6df-5c896694c5-qf4zs\" (UID: \"0b4dec75-b53b-4e05-82eb-553ac25bee7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7ca6df-5c896694c5-qf4zs" Sep 11 18:35:37.475792 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:37.475761 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n9gw\" (UniqueName: \"kubernetes.io/projected/0b4dec75-b53b-4e05-82eb-553ac25bee7d-kube-api-access-6n9gw\") pod \"maas-api-e2e-shared-a-w3-7ca6df-5c896694c5-qf4zs\" (UID: \"0b4dec75-b53b-4e05-82eb-553ac25bee7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7ca6df-5c896694c5-qf4zs" Sep 11 18:35:37.968170 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:37.968127 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b4dec75-b53b-4e05-82eb-553ac25bee7d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-7ca6df-5c896694c5-qf4zs\" (UID: \"0b4dec75-b53b-4e05-82eb-553ac25bee7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7ca6df-5c896694c5-qf4zs" Sep 11 18:35:37.970838 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:37.970806 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b4dec75-b53b-4e05-82eb-553ac25bee7d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-7ca6df-5c896694c5-qf4zs\" (UID: \"0b4dec75-b53b-4e05-82eb-553ac25bee7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7ca6df-5c896694c5-qf4zs" Sep 11 18:35:38.027120 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:38.027082 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-7ca6df-6cb6cb5765-c86n8"] Sep 11 18:35:38.027591 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:38.027576 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e9b9a66-30ad-4d94-95aa-d09ee8eea072" containerName="istio-proxy" Sep 11 18:35:38.027635 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:38.027593 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e9b9a66-30ad-4d94-95aa-d09ee8eea072" containerName="istio-proxy" Sep 11 18:35:38.027691 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:38.027682 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e9b9a66-30ad-4d94-95aa-d09ee8eea072" containerName="istio-proxy" Sep 11 18:35:38.030103 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:38.030072 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7ca6df-6cb6cb5765-c86n8" Sep 11 18:35:38.032654 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:38.032632 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-7ca6df-dockercfg-lpb48\"" Sep 11 18:35:38.032807 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:38.032787 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-7ca6df\"" Sep 11 18:35:38.041235 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:38.041206 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-7ca6df-6cb6cb5765-c86n8"] Sep 11 18:35:38.069031 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:38.068991 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ec34a6b-f340-4743-a761-b5327b9e7516-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-7ca6df-6cb6cb5765-c86n8\" (UID: \"0ec34a6b-f340-4743-a761-b5327b9e7516\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7ca6df-6cb6cb5765-c86n8" Sep 11 18:35:38.069220 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:38.069071 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fxh8\" (UniqueName: \"kubernetes.io/projected/0ec34a6b-f340-4743-a761-b5327b9e7516-kube-api-access-9fxh8\") pod \"payload-processing-e2e-shared-a-w3-7ca6df-6cb6cb5765-c86n8\" (UID: \"0ec34a6b-f340-4743-a761-b5327b9e7516\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7ca6df-6cb6cb5765-c86n8" Sep 11 18:35:38.170611 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:38.170568 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ec34a6b-f340-4743-a761-b5327b9e7516-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-7ca6df-6cb6cb5765-c86n8\" (UID: \"0ec34a6b-f340-4743-a761-b5327b9e7516\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7ca6df-6cb6cb5765-c86n8" Sep 11 18:35:38.170797 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:38.170679 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fxh8\" (UniqueName: \"kubernetes.io/projected/0ec34a6b-f340-4743-a761-b5327b9e7516-kube-api-access-9fxh8\") pod \"payload-processing-e2e-shared-a-w3-7ca6df-6cb6cb5765-c86n8\" (UID: \"0ec34a6b-f340-4743-a761-b5327b9e7516\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7ca6df-6cb6cb5765-c86n8" Sep 11 18:35:38.171412 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:38.171387 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ec34a6b-f340-4743-a761-b5327b9e7516-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-7ca6df-6cb6cb5765-c86n8\" (UID: \"0ec34a6b-f340-4743-a761-b5327b9e7516\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7ca6df-6cb6cb5765-c86n8" Sep 11 18:35:38.181552 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:38.181530 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fxh8\" (UniqueName: \"kubernetes.io/projected/0ec34a6b-f340-4743-a761-b5327b9e7516-kube-api-access-9fxh8\") pod \"payload-processing-e2e-shared-a-w3-7ca6df-6cb6cb5765-c86n8\" (UID: \"0ec34a6b-f340-4743-a761-b5327b9e7516\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7ca6df-6cb6cb5765-c86n8" Sep 11 18:35:38.250903 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:38.250816 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7ca6df-5c896694c5-qf4zs" Sep 11 18:35:38.342782 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:38.342635 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7ca6df-6cb6cb5765-c86n8" Sep 11 18:35:38.527639 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:38.527607 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7ca6df-5c896694c5-qf4zs"] Sep 11 18:35:38.528378 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:35:38.528343 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b4dec75_b53b_4e05_82eb_553ac25bee7d.slice/crio-12214b2008e160b80bcb99cc0ca9bd85b2571d1890d69249a5f707b645922a93 WatchSource:0}: Error finding container 12214b2008e160b80bcb99cc0ca9bd85b2571d1890d69249a5f707b645922a93: Status 404 returned error can't find the container with id 12214b2008e160b80bcb99cc0ca9bd85b2571d1890d69249a5f707b645922a93 Sep 11 18:35:38.621166 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:38.621125 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-7ca6df-6cb6cb5765-c86n8"] Sep 11 18:35:38.621891 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:35:38.621860 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ec34a6b_f340_4743_a761_b5327b9e7516.slice/crio-da99cbb367adac7c3987b522cafe1e84e8795b224b453232568749b6f222a39e WatchSource:0}: Error finding container da99cbb367adac7c3987b522cafe1e84e8795b224b453232568749b6f222a39e: Status 404 returned error can't find the container with id da99cbb367adac7c3987b522cafe1e84e8795b224b453232568749b6f222a39e Sep 11 18:35:38.869040 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:38.868928 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7ca6df-6cb6cb5765-c86n8" event={"ID":"0ec34a6b-f340-4743-a761-b5327b9e7516","Type":"ContainerStarted","Data":"7047447bfa9db574277d479bc02bd507aaacfada7df24f1f94ab4851bba3d70f"} Sep 11 18:35:38.869244 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:38.869086 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7ca6df-6cb6cb5765-c86n8" Sep 11 18:35:38.869244 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:38.869133 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7ca6df-6cb6cb5765-c86n8" event={"ID":"0ec34a6b-f340-4743-a761-b5327b9e7516","Type":"ContainerStarted","Data":"da99cbb367adac7c3987b522cafe1e84e8795b224b453232568749b6f222a39e"} Sep 11 18:35:38.870355 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:38.870331 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7ca6df-5c896694c5-qf4zs" event={"ID":"0b4dec75-b53b-4e05-82eb-553ac25bee7d","Type":"ContainerStarted","Data":"12214b2008e160b80bcb99cc0ca9bd85b2571d1890d69249a5f707b645922a93"} Sep 11 18:35:38.887695 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:38.887634 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7ca6df-6cb6cb5765-c86n8" podStartSLOduration=1.887618905 podStartE2EDuration="1.887618905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:35:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:35:38.883809636 +0000 UTC m=+1220.380827391" watchObservedRunningTime="2026-09-11 18:35:38.887618905 +0000 UTC m=+1220.384636658" Sep 11 18:35:41.906335 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:41.906300 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7ca6df-5c896694c5-qf4zs" event={"ID":"0b4dec75-b53b-4e05-82eb-553ac25bee7d","Type":"ContainerStarted","Data":"a0aa8bb0edadd64029c0a08152571f20b5498c55317088b468c82cb13d8f67dc"} Sep 11 18:35:41.906735 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:41.906442 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7ca6df-5c896694c5-qf4zs" Sep 11 18:35:41.922692 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:41.922634 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7ca6df-5c896694c5-qf4zs" podStartSLOduration=2.61222082 podStartE2EDuration="4.922617461s" totalImagesPullingTime="2.310396641s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:35:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:35:41.922298898 +0000 UTC m=+1223.419316645" watchObservedRunningTime="2026-09-11 18:35:41.922617461 +0000 UTC m=+1223.419635214" Sep 11 18:35:47.918741 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:47.918707 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7ca6df-5c896694c5-qf4zs" Sep 11 18:35:49.889611 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:35:49.889577 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7ca6df-6cb6cb5765-c86n8" Sep 11 18:36:12.945934 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:36:12.945856 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-711a34-6689bb58bb-d58vr"] Sep 11 18:36:12.950402 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:36:12.950379 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-711a34-6689bb58bb-d58vr" Sep 11 18:36:12.952752 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:36:12.952727 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-711a34-serving-cert\"" Sep 11 18:36:12.964504 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:36:12.964472 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-711a34-6689bb58bb-d58vr"] Sep 11 18:36:13.051538 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:36:13.051496 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbe7734f-b586-494a-a1d2-30adae630060-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-711a34-6689bb58bb-d58vr\" (UID: \"bbe7734f-b586-494a-a1d2-30adae630060\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-711a34-6689bb58bb-d58vr" Sep 11 18:36:13.051744 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:36:13.051632 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbe7734f-b586-494a-a1d2-30adae630060-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-711a34-6689bb58bb-d58vr\" (UID: \"bbe7734f-b586-494a-a1d2-30adae630060\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-711a34-6689bb58bb-d58vr" Sep 11 18:36:13.051744 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:36:13.051663 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sfmm\" (UniqueName: \"kubernetes.io/projected/bbe7734f-b586-494a-a1d2-30adae630060-kube-api-access-5sfmm\") pod \"maas-api-e2e-shared-b-w3-711a34-6689bb58bb-d58vr\" (UID: \"bbe7734f-b586-494a-a1d2-30adae630060\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-711a34-6689bb58bb-d58vr" Sep 11 18:36:13.051841 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:36:13.051792 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbe7734f-b586-494a-a1d2-30adae630060-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-711a34-6689bb58bb-d58vr\" (UID: \"bbe7734f-b586-494a-a1d2-30adae630060\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-711a34-6689bb58bb-d58vr" Sep 11 18:36:13.153205 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:36:13.153168 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbe7734f-b586-494a-a1d2-30adae630060-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-711a34-6689bb58bb-d58vr\" (UID: \"bbe7734f-b586-494a-a1d2-30adae630060\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-711a34-6689bb58bb-d58vr" Sep 11 18:36:13.153411 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:36:13.153216 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbe7734f-b586-494a-a1d2-30adae630060-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-711a34-6689bb58bb-d58vr\" (UID: \"bbe7734f-b586-494a-a1d2-30adae630060\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-711a34-6689bb58bb-d58vr" Sep 11 18:36:13.153411 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:36:13.153286 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbe7734f-b586-494a-a1d2-30adae630060-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-711a34-6689bb58bb-d58vr\" (UID: \"bbe7734f-b586-494a-a1d2-30adae630060\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-711a34-6689bb58bb-d58vr" Sep 11 18:36:13.153411 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:36:13.153301 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sfmm\" (UniqueName: \"kubernetes.io/projected/bbe7734f-b586-494a-a1d2-30adae630060-kube-api-access-5sfmm\") pod \"maas-api-e2e-shared-b-w3-711a34-6689bb58bb-d58vr\" (UID: \"bbe7734f-b586-494a-a1d2-30adae630060\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-711a34-6689bb58bb-d58vr" Sep 11 18:36:13.153411 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:36:13.153397 2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-711a34-serving-cert: secret "maas-api-e2e-shared-b-w3-711a34-serving-cert" not found Sep 11 18:36:13.153653 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:36:13.153473 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbe7734f-b586-494a-a1d2-30adae630060-maas-api-tls podName:bbe7734f-b586-494a-a1d2-30adae630060 nodeName:}" failed. No retries permitted until 2026-09-11 18:36:13.653450413 +0000 UTC m=+1255.150468145 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bbe7734f-b586-494a-a1d2-30adae630060-maas-api-tls") pod "maas-api-e2e-shared-b-w3-711a34-6689bb58bb-d58vr" (UID: "bbe7734f-b586-494a-a1d2-30adae630060") : secret "maas-api-e2e-shared-b-w3-711a34-serving-cert" not found Sep 11 18:36:13.154011 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:36:13.153992 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbe7734f-b586-494a-a1d2-30adae630060-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-711a34-6689bb58bb-d58vr\" (UID: \"bbe7734f-b586-494a-a1d2-30adae630060\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-711a34-6689bb58bb-d58vr" Sep 11 18:36:13.155655 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:36:13.155630 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbe7734f-b586-494a-a1d2-30adae630060-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-711a34-6689bb58bb-d58vr\" (UID: \"bbe7734f-b586-494a-a1d2-30adae630060\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-711a34-6689bb58bb-d58vr" Sep 11 18:36:13.161965 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:36:13.161938 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sfmm\" (UniqueName: \"kubernetes.io/projected/bbe7734f-b586-494a-a1d2-30adae630060-kube-api-access-5sfmm\") pod \"maas-api-e2e-shared-b-w3-711a34-6689bb58bb-d58vr\" (UID: \"bbe7734f-b586-494a-a1d2-30adae630060\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-711a34-6689bb58bb-d58vr" Sep 11 18:36:13.659111 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:36:13.659049 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbe7734f-b586-494a-a1d2-30adae630060-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-711a34-6689bb58bb-d58vr\" (UID: \"bbe7734f-b586-494a-a1d2-30adae630060\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-711a34-6689bb58bb-d58vr" Sep 11 18:36:13.661551 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:36:13.661524 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbe7734f-b586-494a-a1d2-30adae630060-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-711a34-6689bb58bb-d58vr\" (UID: \"bbe7734f-b586-494a-a1d2-30adae630060\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-711a34-6689bb58bb-d58vr" Sep 11 18:36:13.863895 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:36:13.863844 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-711a34-6689bb58bb-d58vr" Sep 11 18:36:14.141165 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:36:14.141134 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-711a34-6689bb58bb-d58vr"] Sep 11 18:36:14.142386 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:36:14.142356 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbe7734f_b586_494a_a1d2_30adae630060.slice/crio-190249f9cbfc79f057e7dc19a8619b251f834a2256c54f170ec9c57eee71592c WatchSource:0}: Error finding container 190249f9cbfc79f057e7dc19a8619b251f834a2256c54f170ec9c57eee71592c: Status 404 returned error can't find the container with id 190249f9cbfc79f057e7dc19a8619b251f834a2256c54f170ec9c57eee71592c Sep 11 18:36:15.144247 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:36:15.144212 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-711a34-6689bb58bb-d58vr" event={"ID":"bbe7734f-b586-494a-a1d2-30adae630060","Type":"ContainerStarted","Data":"d1bebd9c348a1bb9b7ab451a4dafe27b9f3c379740ede00ce1f989157df4438c"} Sep 11 18:36:15.144744 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:36:15.144270 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-711a34-6689bb58bb-d58vr" event={"ID":"bbe7734f-b586-494a-a1d2-30adae630060","Type":"ContainerStarted","Data":"190249f9cbfc79f057e7dc19a8619b251f834a2256c54f170ec9c57eee71592c"} Sep 11 18:36:15.144744 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:36:15.144439 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-711a34-6689bb58bb-d58vr" Sep 11 18:36:15.163232 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:36:15.163178 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-711a34-6689bb58bb-d58vr" podStartSLOduration=2.812537791 podStartE2EDuration="3.163160781s" totalImagesPullingTime="350.62299ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:36:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:36:15.161474154 +0000 UTC m=+1256.658491910" watchObservedRunningTime="2026-09-11 18:36:15.163160781 +0000 UTC m=+1256.660178535" Sep 11 18:36:19.549911 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:36:19.549875 2561 scope.go:117] "RemoveContainer" containerID="04e92e3d37eab8ebcd55d40fdbfc31b381cc5bf8082520aacd829499500503b1" Sep 11 18:36:19.562676 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:36:19.562649 2561 scope.go:117] "RemoveContainer" containerID="a63bef399a98a4f6152d8ea5adae6399f6ddfd345155eec1acb8c4ef59bccf4e" Sep 11 18:36:19.572237 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:36:19.572212 2561 scope.go:117] "RemoveContainer" containerID="33458db655c8b15fd1e155b310051e8fb5a491f7197438a843502cbe1e71f843" Sep 11 18:36:19.581023 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:36:19.581000 2561 scope.go:117] "RemoveContainer" containerID="41bb20498fddb7bcd523d9c4d4d9d2e1f8ef58af4dc22726f2b626a4389b0052" Sep 11 18:36:21.157161 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:36:21.157135 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-711a34-6689bb58bb-d58vr" Sep 11 18:37:32.062170 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:37:32.062128 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-18a45f6a-596dcbb4cd-zjpgq"] Sep 11 18:37:32.062691 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:37:32.062422 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-18a45f6a-596dcbb4cd-zjpgq" podUID="05e89f9e-a6a1-4652-b69d-6554cea2e82d" containerName="payload-pre-processing" containerID="cri-o://04d7d5e8421653c8af983d480815433d71af068f9047d85d18c7b5c733ed804a" gracePeriod=30 Sep 11 18:37:32.344763 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:37:32.344738 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-18a45f6a-596dcbb4cd-zjpgq" Sep 11 18:37:32.429764 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:37:32.429729 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05e89f9e-a6a1-4652-b69d-6554cea2e82d-plugins-config-volume\") pod \"05e89f9e-a6a1-4652-b69d-6554cea2e82d\" (UID: \"05e89f9e-a6a1-4652-b69d-6554cea2e82d\") " Sep 11 18:37:32.429966 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:37:32.429815 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7qfm6\" (UniqueName: \"kubernetes.io/projected/05e89f9e-a6a1-4652-b69d-6554cea2e82d-kube-api-access-7qfm6\") pod \"05e89f9e-a6a1-4652-b69d-6554cea2e82d\" (UID: \"05e89f9e-a6a1-4652-b69d-6554cea2e82d\") " Sep 11 18:37:32.430209 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:37:32.430182 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05e89f9e-a6a1-4652-b69d-6554cea2e82d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "05e89f9e-a6a1-4652-b69d-6554cea2e82d" (UID: "05e89f9e-a6a1-4652-b69d-6554cea2e82d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:37:32.430308 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:37:32.430290 2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05e89f9e-a6a1-4652-b69d-6554cea2e82d-plugins-config-volume\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:37:32.432007 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:37:32.431978 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05e89f9e-a6a1-4652-b69d-6554cea2e82d-kube-api-access-7qfm6" (OuterVolumeSpecName: "kube-api-access-7qfm6") pod "05e89f9e-a6a1-4652-b69d-6554cea2e82d" (UID: "05e89f9e-a6a1-4652-b69d-6554cea2e82d"). InnerVolumeSpecName "kube-api-access-7qfm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:37:32.530935 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:37:32.530895 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qfm6\" (UniqueName: \"kubernetes.io/projected/05e89f9e-a6a1-4652-b69d-6554cea2e82d-kube-api-access-7qfm6\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:37:32.700421 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:37:32.700389 2561 generic.go:356] "Generic (PLEG): container finished" podID="05e89f9e-a6a1-4652-b69d-6554cea2e82d" containerID="04d7d5e8421653c8af983d480815433d71af068f9047d85d18c7b5c733ed804a" exitCode=0 Sep 11 18:37:32.700619 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:37:32.700440 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-18a45f6a-596dcbb4cd-zjpgq" Sep 11 18:37:32.700619 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:37:32.700479 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-18a45f6a-596dcbb4cd-zjpgq" event={"ID":"05e89f9e-a6a1-4652-b69d-6554cea2e82d","Type":"ContainerDied","Data":"04d7d5e8421653c8af983d480815433d71af068f9047d85d18c7b5c733ed804a"} Sep 11 18:37:32.700619 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:37:32.700539 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-18a45f6a-596dcbb4cd-zjpgq" event={"ID":"05e89f9e-a6a1-4652-b69d-6554cea2e82d","Type":"ContainerDied","Data":"e941f82201fc505f64afb3444d5f1b807dee0eaed6ceb799abdb82b35d992ad5"} Sep 11 18:37:32.700619 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:37:32.700570 2561 scope.go:117] "RemoveContainer" containerID="04d7d5e8421653c8af983d480815433d71af068f9047d85d18c7b5c733ed804a" Sep 11 18:37:32.710944 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:37:32.710923 2561 scope.go:117] "RemoveContainer" containerID="04d7d5e8421653c8af983d480815433d71af068f9047d85d18c7b5c733ed804a" Sep 11 18:37:32.711252 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:37:32.711230 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04d7d5e8421653c8af983d480815433d71af068f9047d85d18c7b5c733ed804a\": container with ID starting with 04d7d5e8421653c8af983d480815433d71af068f9047d85d18c7b5c733ed804a not found: ID does not exist" containerID="04d7d5e8421653c8af983d480815433d71af068f9047d85d18c7b5c733ed804a" Sep 11 18:37:32.711312 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:37:32.711262 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04d7d5e8421653c8af983d480815433d71af068f9047d85d18c7b5c733ed804a"} err="failed to get container status \"04d7d5e8421653c8af983d480815433d71af068f9047d85d18c7b5c733ed804a\": rpc error: code = NotFound desc = could not find container \"04d7d5e8421653c8af983d480815433d71af068f9047d85d18c7b5c733ed804a\": container with ID starting with 04d7d5e8421653c8af983d480815433d71af068f9047d85d18c7b5c733ed804a not found: ID does not exist" Sep 11 18:37:32.724251 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:37:32.724220 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-18a45f6a-596dcbb4cd-zjpgq"] Sep 11 18:37:32.731496 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:37:32.731467 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-18a45f6a-596dcbb4cd-zjpgq"] Sep 11 18:37:33.140198 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:37:33.140117 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05e89f9e-a6a1-4652-b69d-6554cea2e82d" path="/var/lib/kubelet/pods/05e89f9e-a6a1-4652-b69d-6554cea2e82d/volumes" Sep 11 18:38:39.031903 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:38:39.031865 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj"] Sep 11 18:38:39.032528 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:38:39.032096 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj" podUID="0ae85a40-303c-45f3-9eaf-72b042756d2e" containerName="istio-proxy" containerID="cri-o://4cfe11f0ce5b790cad2706ce6eba3e0a702d0cac1e73afa12d587488aee22392" gracePeriod=30 Sep 11 18:38:44.196242 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:38:44.196208 2561 generic.go:356] "Generic (PLEG): container finished" podID="0ae85a40-303c-45f3-9eaf-72b042756d2e" containerID="4cfe11f0ce5b790cad2706ce6eba3e0a702d0cac1e73afa12d587488aee22392" exitCode=0 Sep 11 18:38:44.196693 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:38:44.196282 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj" event={"ID":"0ae85a40-303c-45f3-9eaf-72b042756d2e","Type":"ContainerDied","Data":"4cfe11f0ce5b790cad2706ce6eba3e0a702d0cac1e73afa12d587488aee22392"} Sep 11 18:38:44.196693 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:38:44.196357 2561 scope.go:117] "RemoveContainer" containerID="401892651b30cf81a0fcf502e6949f755c83625b1ca16db5887caf44496d62ef" Sep 11 18:38:44.291204 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:38:44.291172 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj" Sep 11 18:38:44.373691 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:38:44.373608 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ae85a40-303c-45f3-9eaf-72b042756d2e-workload-socket\") pod \"0ae85a40-303c-45f3-9eaf-72b042756d2e\" (UID: \"0ae85a40-303c-45f3-9eaf-72b042756d2e\") " Sep 11 18:38:44.373691 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:38:44.373670 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ae85a40-303c-45f3-9eaf-72b042756d2e-istio-podinfo\") pod \"0ae85a40-303c-45f3-9eaf-72b042756d2e\" (UID: \"0ae85a40-303c-45f3-9eaf-72b042756d2e\") " Sep 11 18:38:44.373691 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:38:44.373694 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ae85a40-303c-45f3-9eaf-72b042756d2e-istio-token\") pod \"0ae85a40-303c-45f3-9eaf-72b042756d2e\" (UID: \"0ae85a40-303c-45f3-9eaf-72b042756d2e\") " Sep 11 18:38:44.373974 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:38:44.373730 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d7jcw\" (UniqueName: \"kubernetes.io/projected/0ae85a40-303c-45f3-9eaf-72b042756d2e-kube-api-access-d7jcw\") pod \"0ae85a40-303c-45f3-9eaf-72b042756d2e\" (UID: \"0ae85a40-303c-45f3-9eaf-72b042756d2e\") " Sep 11 18:38:44.373974 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:38:44.373758 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ae85a40-303c-45f3-9eaf-72b042756d2e-istiod-ca-cert\") pod \"0ae85a40-303c-45f3-9eaf-72b042756d2e\" (UID: \"0ae85a40-303c-45f3-9eaf-72b042756d2e\") " Sep 11 18:38:44.373974 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:38:44.373889 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ae85a40-303c-45f3-9eaf-72b042756d2e-istio-data\") pod \"0ae85a40-303c-45f3-9eaf-72b042756d2e\" (UID: \"0ae85a40-303c-45f3-9eaf-72b042756d2e\") " Sep 11 18:38:44.373974 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:38:44.373909 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ae85a40-303c-45f3-9eaf-72b042756d2e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0ae85a40-303c-45f3-9eaf-72b042756d2e" (UID: "0ae85a40-303c-45f3-9eaf-72b042756d2e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:38:44.373974 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:38:44.373961 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ae85a40-303c-45f3-9eaf-72b042756d2e-istio-envoy\") pod \"0ae85a40-303c-45f3-9eaf-72b042756d2e\" (UID: \"0ae85a40-303c-45f3-9eaf-72b042756d2e\") " Sep 11 18:38:44.374303 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:38:44.373999 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ae85a40-303c-45f3-9eaf-72b042756d2e-credential-socket\") pod \"0ae85a40-303c-45f3-9eaf-72b042756d2e\" (UID: \"0ae85a40-303c-45f3-9eaf-72b042756d2e\") " Sep 11 18:38:44.374303 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:38:44.374080 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ae85a40-303c-45f3-9eaf-72b042756d2e-workload-certs\") pod \"0ae85a40-303c-45f3-9eaf-72b042756d2e\" (UID: \"0ae85a40-303c-45f3-9eaf-72b042756d2e\") " Sep 11 18:38:44.374303 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:38:44.374145 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ae85a40-303c-45f3-9eaf-72b042756d2e-istio-data" (OuterVolumeSpecName: "istio-data") pod "0ae85a40-303c-45f3-9eaf-72b042756d2e" (UID: "0ae85a40-303c-45f3-9eaf-72b042756d2e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:38:44.374303 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:38:44.374178 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ae85a40-303c-45f3-9eaf-72b042756d2e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0ae85a40-303c-45f3-9eaf-72b042756d2e" (UID: "0ae85a40-303c-45f3-9eaf-72b042756d2e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:38:44.374513 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:38:44.374330 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ae85a40-303c-45f3-9eaf-72b042756d2e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0ae85a40-303c-45f3-9eaf-72b042756d2e" (UID: "0ae85a40-303c-45f3-9eaf-72b042756d2e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:38:44.374513 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:38:44.374497 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ae85a40-303c-45f3-9eaf-72b042756d2e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0ae85a40-303c-45f3-9eaf-72b042756d2e" (UID: "0ae85a40-303c-45f3-9eaf-72b042756d2e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:38:44.374627 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:38:44.374553 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ae85a40-303c-45f3-9eaf-72b042756d2e-istiod-ca-cert\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:38:44.374627 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:38:44.374567 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ae85a40-303c-45f3-9eaf-72b042756d2e-istio-data\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:38:44.374627 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:38:44.374576 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ae85a40-303c-45f3-9eaf-72b042756d2e-credential-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:38:44.374627 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:38:44.374585 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ae85a40-303c-45f3-9eaf-72b042756d2e-workload-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:38:44.376010 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:38:44.375982 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ae85a40-303c-45f3-9eaf-72b042756d2e-kube-api-access-d7jcw" (OuterVolumeSpecName: "kube-api-access-d7jcw") pod "0ae85a40-303c-45f3-9eaf-72b042756d2e" (UID: "0ae85a40-303c-45f3-9eaf-72b042756d2e"). InnerVolumeSpecName "kube-api-access-d7jcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:38:44.376160 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:38:44.376021 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ae85a40-303c-45f3-9eaf-72b042756d2e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0ae85a40-303c-45f3-9eaf-72b042756d2e" (UID: "0ae85a40-303c-45f3-9eaf-72b042756d2e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 18:38:44.376402 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:38:44.376378 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ae85a40-303c-45f3-9eaf-72b042756d2e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0ae85a40-303c-45f3-9eaf-72b042756d2e" (UID: "0ae85a40-303c-45f3-9eaf-72b042756d2e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:38:44.376507 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:38:44.376400 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ae85a40-303c-45f3-9eaf-72b042756d2e-istio-token" (OuterVolumeSpecName: "istio-token") pod "0ae85a40-303c-45f3-9eaf-72b042756d2e" (UID: "0ae85a40-303c-45f3-9eaf-72b042756d2e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:38:44.475690 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:38:44.475654 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ae85a40-303c-45f3-9eaf-72b042756d2e-istio-envoy\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:38:44.475690 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:38:44.475684 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ae85a40-303c-45f3-9eaf-72b042756d2e-workload-certs\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:38:44.475690 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:38:44.475693 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ae85a40-303c-45f3-9eaf-72b042756d2e-istio-podinfo\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:38:44.475690 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:38:44.475701 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ae85a40-303c-45f3-9eaf-72b042756d2e-istio-token\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:38:44.475956 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:38:44.475710 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7jcw\" (UniqueName: \"kubernetes.io/projected/0ae85a40-303c-45f3-9eaf-72b042756d2e-kube-api-access-d7jcw\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:38:45.220071 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:38:45.220015 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj" Sep 11 18:38:45.220071 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:38:45.220026 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj" event={"ID":"0ae85a40-303c-45f3-9eaf-72b042756d2e","Type":"ContainerDied","Data":"0fd98b86211f19ca25d221c858378d30618f23bb9beb47cd3d8b2f566e351eed"} Sep 11 18:38:45.220624 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:38:45.220154 2561 scope.go:117] "RemoveContainer" containerID="4cfe11f0ce5b790cad2706ce6eba3e0a702d0cac1e73afa12d587488aee22392" Sep 11 18:38:45.240902 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:38:45.240841 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj"] Sep 11 18:38:45.244527 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:38:45.244497 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-18a45f6a-openshift-default-78d446b676-qggsj"] Sep 11 18:38:47.140064 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:38:47.140012 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ae85a40-303c-45f3-9eaf-72b042756d2e" path="/var/lib/kubelet/pods/0ae85a40-303c-45f3-9eaf-72b042756d2e/volumes" Sep 11 18:39:19.655546 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:19.655463 2561 scope.go:117] "RemoveContainer" containerID="88025ca099ac755cd0847912032990f9d2b03b80d9007f76f3bb3bdd3072d374" Sep 11 18:39:19.665037 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:19.665016 2561 scope.go:117] "RemoveContainer" containerID="e4af4faac8d565388591073cb505cfbde84362d61595a2b17c10ebc7e04913c7" Sep 11 18:39:24.753283 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:24.753249 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-efa8b9/embedding-w0-efa8b9-kserve-6d8f694465-l5drh"] Sep 11 18:39:24.753725 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:24.753711 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05e89f9e-a6a1-4652-b69d-6554cea2e82d" containerName="payload-pre-processing" Sep 11 18:39:24.753725 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:24.753727 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="05e89f9e-a6a1-4652-b69d-6554cea2e82d" containerName="payload-pre-processing" Sep 11 18:39:24.753812 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:24.753739 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ae85a40-303c-45f3-9eaf-72b042756d2e" containerName="istio-proxy" Sep 11 18:39:24.753812 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:24.753748 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ae85a40-303c-45f3-9eaf-72b042756d2e" containerName="istio-proxy" Sep 11 18:39:24.753812 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:24.753771 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ae85a40-303c-45f3-9eaf-72b042756d2e" containerName="istio-proxy" Sep 11 18:39:24.753812 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:24.753778 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ae85a40-303c-45f3-9eaf-72b042756d2e" containerName="istio-proxy" Sep 11 18:39:24.753949 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:24.753847 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="05e89f9e-a6a1-4652-b69d-6554cea2e82d" containerName="payload-pre-processing" Sep 11 18:39:24.753949 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:24.753855 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ae85a40-303c-45f3-9eaf-72b042756d2e" containerName="istio-proxy" Sep 11 18:39:24.753949 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:24.753861 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ae85a40-303c-45f3-9eaf-72b042756d2e" containerName="istio-proxy" Sep 11 18:39:24.758492 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:24.758473 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-efa8b9/embedding-w0-efa8b9-kserve-6d8f694465-l5drh" Sep 11 18:39:24.761554 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:24.761523 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-efa8b9\"/\"default-dockercfg-gn54h\"" Sep 11 18:39:24.761722 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:24.761602 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-efa8b9\"/\"embedding-w0-efa8b9-kserve-self-signed-certs\"" Sep 11 18:39:24.761722 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:24.761645 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-efa8b9\"/\"kube-root-ca.crt\"" Sep 11 18:39:24.761891 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:24.761738 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-efa8b9\"/\"openshift-service-ca.crt\"" Sep 11 18:39:24.766858 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:24.766832 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-efa8b9/embedding-w0-efa8b9-kserve-6d8f694465-l5drh"] Sep 11 18:39:24.798542 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:24.798511 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e8a1657-b905-4add-84b2-5549f720a1d2-model-cache\") pod \"embedding-w0-efa8b9-kserve-6d8f694465-l5drh\" (UID: \"5e8a1657-b905-4add-84b2-5549f720a1d2\") " pod="e2e-models-e2e-worker-w0-efa8b9/embedding-w0-efa8b9-kserve-6d8f694465-l5drh" Sep 11 18:39:24.798732 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:24.798565 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmjtx\" (UniqueName: \"kubernetes.io/projected/5e8a1657-b905-4add-84b2-5549f720a1d2-kube-api-access-pmjtx\") pod \"embedding-w0-efa8b9-kserve-6d8f694465-l5drh\" (UID: \"5e8a1657-b905-4add-84b2-5549f720a1d2\") " pod="e2e-models-e2e-worker-w0-efa8b9/embedding-w0-efa8b9-kserve-6d8f694465-l5drh" Sep 11 18:39:24.798802 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:24.798722 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e8a1657-b905-4add-84b2-5549f720a1d2-dshm\") pod \"embedding-w0-efa8b9-kserve-6d8f694465-l5drh\" (UID: \"5e8a1657-b905-4add-84b2-5549f720a1d2\") " pod="e2e-models-e2e-worker-w0-efa8b9/embedding-w0-efa8b9-kserve-6d8f694465-l5drh" Sep 11 18:39:24.798802 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:24.798775 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e8a1657-b905-4add-84b2-5549f720a1d2-tmp-dir\") pod \"embedding-w0-efa8b9-kserve-6d8f694465-l5drh\" (UID: \"5e8a1657-b905-4add-84b2-5549f720a1d2\") " pod="e2e-models-e2e-worker-w0-efa8b9/embedding-w0-efa8b9-kserve-6d8f694465-l5drh" Sep 11 18:39:24.798929 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:24.798909 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e8a1657-b905-4add-84b2-5549f720a1d2-home\") pod \"embedding-w0-efa8b9-kserve-6d8f694465-l5drh\" (UID: \"5e8a1657-b905-4add-84b2-5549f720a1d2\") " pod="e2e-models-e2e-worker-w0-efa8b9/embedding-w0-efa8b9-kserve-6d8f694465-l5drh" Sep 11 18:39:24.798985 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:24.798955 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e8a1657-b905-4add-84b2-5549f720a1d2-tls-certs\") pod \"embedding-w0-efa8b9-kserve-6d8f694465-l5drh\" (UID: \"5e8a1657-b905-4add-84b2-5549f720a1d2\") " pod="e2e-models-e2e-worker-w0-efa8b9/embedding-w0-efa8b9-kserve-6d8f694465-l5drh" Sep 11 18:39:24.899568 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:24.899525 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e8a1657-b905-4add-84b2-5549f720a1d2-home\") pod \"embedding-w0-efa8b9-kserve-6d8f694465-l5drh\" (UID: \"5e8a1657-b905-4add-84b2-5549f720a1d2\") " pod="e2e-models-e2e-worker-w0-efa8b9/embedding-w0-efa8b9-kserve-6d8f694465-l5drh" Sep 11 18:39:24.899760 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:24.899577 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e8a1657-b905-4add-84b2-5549f720a1d2-tls-certs\") pod \"embedding-w0-efa8b9-kserve-6d8f694465-l5drh\" (UID: \"5e8a1657-b905-4add-84b2-5549f720a1d2\") " pod="e2e-models-e2e-worker-w0-efa8b9/embedding-w0-efa8b9-kserve-6d8f694465-l5drh" Sep 11 18:39:24.899760 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:24.899634 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e8a1657-b905-4add-84b2-5549f720a1d2-model-cache\") pod \"embedding-w0-efa8b9-kserve-6d8f694465-l5drh\" (UID: \"5e8a1657-b905-4add-84b2-5549f720a1d2\") " pod="e2e-models-e2e-worker-w0-efa8b9/embedding-w0-efa8b9-kserve-6d8f694465-l5drh" Sep 11 18:39:24.899760 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:24.899672 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmjtx\" (UniqueName: \"kubernetes.io/projected/5e8a1657-b905-4add-84b2-5549f720a1d2-kube-api-access-pmjtx\") pod \"embedding-w0-efa8b9-kserve-6d8f694465-l5drh\" (UID: \"5e8a1657-b905-4add-84b2-5549f720a1d2\") " pod="e2e-models-e2e-worker-w0-efa8b9/embedding-w0-efa8b9-kserve-6d8f694465-l5drh" Sep 11 18:39:24.899760 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:24.899724 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e8a1657-b905-4add-84b2-5549f720a1d2-dshm\") pod \"embedding-w0-efa8b9-kserve-6d8f694465-l5drh\" (UID: \"5e8a1657-b905-4add-84b2-5549f720a1d2\") " pod="e2e-models-e2e-worker-w0-efa8b9/embedding-w0-efa8b9-kserve-6d8f694465-l5drh" Sep 11 18:39:24.899760 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:24.899750 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e8a1657-b905-4add-84b2-5549f720a1d2-tmp-dir\") pod \"embedding-w0-efa8b9-kserve-6d8f694465-l5drh\" (UID: \"5e8a1657-b905-4add-84b2-5549f720a1d2\") " pod="e2e-models-e2e-worker-w0-efa8b9/embedding-w0-efa8b9-kserve-6d8f694465-l5drh" Sep 11 18:39:24.900039 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:24.899960 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e8a1657-b905-4add-84b2-5549f720a1d2-home\") pod \"embedding-w0-efa8b9-kserve-6d8f694465-l5drh\" (UID: \"5e8a1657-b905-4add-84b2-5549f720a1d2\") " pod="e2e-models-e2e-worker-w0-efa8b9/embedding-w0-efa8b9-kserve-6d8f694465-l5drh" Sep 11 18:39:24.900039 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:24.900005 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e8a1657-b905-4add-84b2-5549f720a1d2-model-cache\") pod \"embedding-w0-efa8b9-kserve-6d8f694465-l5drh\" (UID: \"5e8a1657-b905-4add-84b2-5549f720a1d2\") " pod="e2e-models-e2e-worker-w0-efa8b9/embedding-w0-efa8b9-kserve-6d8f694465-l5drh" Sep 11 18:39:24.900201 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:24.900165 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e8a1657-b905-4add-84b2-5549f720a1d2-tmp-dir\") pod \"embedding-w0-efa8b9-kserve-6d8f694465-l5drh\" (UID: \"5e8a1657-b905-4add-84b2-5549f720a1d2\") " pod="e2e-models-e2e-worker-w0-efa8b9/embedding-w0-efa8b9-kserve-6d8f694465-l5drh" Sep 11 18:39:24.901908 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:24.901884 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e8a1657-b905-4add-84b2-5549f720a1d2-dshm\") pod \"embedding-w0-efa8b9-kserve-6d8f694465-l5drh\" (UID: \"5e8a1657-b905-4add-84b2-5549f720a1d2\") " pod="e2e-models-e2e-worker-w0-efa8b9/embedding-w0-efa8b9-kserve-6d8f694465-l5drh" Sep 11 18:39:24.902257 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:24.902237 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e8a1657-b905-4add-84b2-5549f720a1d2-tls-certs\") pod \"embedding-w0-efa8b9-kserve-6d8f694465-l5drh\" (UID: \"5e8a1657-b905-4add-84b2-5549f720a1d2\") " pod="e2e-models-e2e-worker-w0-efa8b9/embedding-w0-efa8b9-kserve-6d8f694465-l5drh" Sep 11 18:39:24.907931 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:24.907908 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmjtx\" (UniqueName: \"kubernetes.io/projected/5e8a1657-b905-4add-84b2-5549f720a1d2-kube-api-access-pmjtx\") pod \"embedding-w0-efa8b9-kserve-6d8f694465-l5drh\" (UID: \"5e8a1657-b905-4add-84b2-5549f720a1d2\") " pod="e2e-models-e2e-worker-w0-efa8b9/embedding-w0-efa8b9-kserve-6d8f694465-l5drh" Sep 11 18:39:25.070191 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:25.070098 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-efa8b9/embedding-w0-efa8b9-kserve-6d8f694465-l5drh" Sep 11 18:39:25.348072 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:25.348029 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-efa8b9/embedding-w0-efa8b9-kserve-6d8f694465-l5drh"] Sep 11 18:39:25.350832 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:39:25.350779 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5e8a1657_b905_4add_84b2_5549f720a1d2.slice/crio-d35e3cb5ce2eb6dac4e4e5ef03031e8e68309966f2c1cda65a7a40693dd6dcde WatchSource:0}: Error finding container d35e3cb5ce2eb6dac4e4e5ef03031e8e68309966f2c1cda65a7a40693dd6dcde: Status 404 returned error can't find the container with id d35e3cb5ce2eb6dac4e4e5ef03031e8e68309966f2c1cda65a7a40693dd6dcde Sep 11 18:39:25.505864 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:25.505713 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-efa8b9/embedding-w0-efa8b9-kserve-6d8f694465-l5drh" event={"ID":"5e8a1657-b905-4add-84b2-5549f720a1d2","Type":"ContainerStarted","Data":"f73ee44416f4ffcbe319e6d08535687099e4d7ecdec454822198132e4a3a042f"} Sep 11 18:39:25.505864 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:25.505756 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-efa8b9/embedding-w0-efa8b9-kserve-6d8f694465-l5drh" event={"ID":"5e8a1657-b905-4add-84b2-5549f720a1d2","Type":"ContainerStarted","Data":"d35e3cb5ce2eb6dac4e4e5ef03031e8e68309966f2c1cda65a7a40693dd6dcde"} Sep 11 18:39:25.528864 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:25.528807 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-efa8b9/embedding-w0-efa8b9-kserve-6d8f694465-l5drh" podStartSLOduration=1.528790069 podStartE2EDuration="1.528790069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:39:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:39:25.52575167 +0000 UTC m=+1447.022769441" watchObservedRunningTime="2026-09-11 18:39:25.528790069 +0000 UTC m=+1447.025807822" Sep 11 18:39:33.832799 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:33.832755 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr"] Sep 11 18:39:33.836944 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:33.836924 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr" Sep 11 18:39:33.839318 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:33.839289 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-c1409b-openshift-default-dockercfg-tt5r7\"" Sep 11 18:39:33.849149 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:33.849120 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr"] Sep 11 18:39:33.890609 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:33.890559 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2df4550-54e5-42d1-9257-077917d8d1b3-workload-certs\") pod \"e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr\" (UID: \"e2df4550-54e5-42d1-9257-077917d8d1b3\") " pod="openshift-ingress/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr" Sep 11 18:39:33.890609 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:33.890609 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pks6r\" (UniqueName: \"kubernetes.io/projected/e2df4550-54e5-42d1-9257-077917d8d1b3-kube-api-access-pks6r\") pod \"e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr\" (UID: \"e2df4550-54e5-42d1-9257-077917d8d1b3\") " pod="openshift-ingress/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr" Sep 11 18:39:33.890856 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:33.890647 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2df4550-54e5-42d1-9257-077917d8d1b3-credential-socket\") pod \"e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr\" (UID: \"e2df4550-54e5-42d1-9257-077917d8d1b3\") " pod="openshift-ingress/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr" Sep 11 18:39:33.890856 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:33.890701 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2df4550-54e5-42d1-9257-077917d8d1b3-istio-data\") pod \"e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr\" (UID: \"e2df4550-54e5-42d1-9257-077917d8d1b3\") " pod="openshift-ingress/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr" Sep 11 18:39:33.890856 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:33.890740 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2df4550-54e5-42d1-9257-077917d8d1b3-workload-socket\") pod \"e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr\" (UID: \"e2df4550-54e5-42d1-9257-077917d8d1b3\") " pod="openshift-ingress/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr" Sep 11 18:39:33.890856 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:33.890763 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2df4550-54e5-42d1-9257-077917d8d1b3-istio-podinfo\") pod \"e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr\" (UID: \"e2df4550-54e5-42d1-9257-077917d8d1b3\") " pod="openshift-ingress/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr" Sep 11 18:39:33.890856 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:33.890807 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2df4550-54e5-42d1-9257-077917d8d1b3-istiod-ca-cert\") pod \"e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr\" (UID: \"e2df4550-54e5-42d1-9257-077917d8d1b3\") " pod="openshift-ingress/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr" Sep 11 18:39:33.890856 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:33.890831 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2df4550-54e5-42d1-9257-077917d8d1b3-istio-envoy\") pod \"e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr\" (UID: \"e2df4550-54e5-42d1-9257-077917d8d1b3\") " pod="openshift-ingress/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr" Sep 11 18:39:33.891196 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:33.890862 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2df4550-54e5-42d1-9257-077917d8d1b3-istio-token\") pod \"e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr\" (UID: \"e2df4550-54e5-42d1-9257-077917d8d1b3\") " pod="openshift-ingress/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr" Sep 11 18:39:33.992369 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:33.992332 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2df4550-54e5-42d1-9257-077917d8d1b3-istiod-ca-cert\") pod \"e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr\" (UID: \"e2df4550-54e5-42d1-9257-077917d8d1b3\") " pod="openshift-ingress/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr" Sep 11 18:39:33.992543 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:33.992378 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2df4550-54e5-42d1-9257-077917d8d1b3-istio-envoy\") pod \"e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr\" (UID: \"e2df4550-54e5-42d1-9257-077917d8d1b3\") " pod="openshift-ingress/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr" Sep 11 18:39:33.992543 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:33.992499 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2df4550-54e5-42d1-9257-077917d8d1b3-istio-token\") pod \"e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr\" (UID: \"e2df4550-54e5-42d1-9257-077917d8d1b3\") " pod="openshift-ingress/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr" Sep 11 18:39:33.992666 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:33.992596 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2df4550-54e5-42d1-9257-077917d8d1b3-workload-certs\") pod \"e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr\" (UID: \"e2df4550-54e5-42d1-9257-077917d8d1b3\") " pod="openshift-ingress/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr" Sep 11 18:39:33.992666 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:33.992620 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pks6r\" (UniqueName: \"kubernetes.io/projected/e2df4550-54e5-42d1-9257-077917d8d1b3-kube-api-access-pks6r\") pod \"e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr\" (UID: \"e2df4550-54e5-42d1-9257-077917d8d1b3\") " pod="openshift-ingress/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr" Sep 11 18:39:33.992759 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:33.992672 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2df4550-54e5-42d1-9257-077917d8d1b3-credential-socket\") pod \"e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr\" (UID: \"e2df4550-54e5-42d1-9257-077917d8d1b3\") " pod="openshift-ingress/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr" Sep 11 18:39:33.992759 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:33.992715 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2df4550-54e5-42d1-9257-077917d8d1b3-istio-data\") pod \"e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr\" (UID: \"e2df4550-54e5-42d1-9257-077917d8d1b3\") " pod="openshift-ingress/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr" Sep 11 18:39:33.992865 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:33.992769 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2df4550-54e5-42d1-9257-077917d8d1b3-workload-socket\") pod \"e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr\" (UID: \"e2df4550-54e5-42d1-9257-077917d8d1b3\") " pod="openshift-ingress/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr" Sep 11 18:39:33.992865 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:33.992800 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2df4550-54e5-42d1-9257-077917d8d1b3-istio-podinfo\") pod \"e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr\" (UID: \"e2df4550-54e5-42d1-9257-077917d8d1b3\") " pod="openshift-ingress/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr" Sep 11 18:39:33.992994 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:33.992963 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2df4550-54e5-42d1-9257-077917d8d1b3-workload-certs\") pod \"e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr\" (UID: \"e2df4550-54e5-42d1-9257-077917d8d1b3\") " pod="openshift-ingress/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr" Sep 11 18:39:33.993260 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:33.993236 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2df4550-54e5-42d1-9257-077917d8d1b3-istiod-ca-cert\") pod \"e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr\" (UID: \"e2df4550-54e5-42d1-9257-077917d8d1b3\") " pod="openshift-ingress/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr" Sep 11 18:39:33.993424 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:33.993237 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2df4550-54e5-42d1-9257-077917d8d1b3-workload-socket\") pod \"e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr\" (UID: \"e2df4550-54e5-42d1-9257-077917d8d1b3\") " pod="openshift-ingress/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr" Sep 11 18:39:33.993495 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:33.993268 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2df4550-54e5-42d1-9257-077917d8d1b3-istio-data\") pod \"e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr\" (UID: \"e2df4550-54e5-42d1-9257-077917d8d1b3\") " pod="openshift-ingress/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr" Sep 11 18:39:33.993495 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:33.993325 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2df4550-54e5-42d1-9257-077917d8d1b3-credential-socket\") pod \"e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr\" (UID: \"e2df4550-54e5-42d1-9257-077917d8d1b3\") " pod="openshift-ingress/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr" Sep 11 18:39:33.995091 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:33.995069 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2df4550-54e5-42d1-9257-077917d8d1b3-istio-envoy\") pod \"e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr\" (UID: \"e2df4550-54e5-42d1-9257-077917d8d1b3\") " pod="openshift-ingress/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr" Sep 11 18:39:33.995208 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:33.995177 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2df4550-54e5-42d1-9257-077917d8d1b3-istio-podinfo\") pod \"e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr\" (UID: \"e2df4550-54e5-42d1-9257-077917d8d1b3\") " pod="openshift-ingress/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr" Sep 11 18:39:34.001318 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:34.001292 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2df4550-54e5-42d1-9257-077917d8d1b3-istio-token\") pod \"e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr\" (UID: \"e2df4550-54e5-42d1-9257-077917d8d1b3\") " pod="openshift-ingress/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr" Sep 11 18:39:34.001752 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:34.001732 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pks6r\" (UniqueName: \"kubernetes.io/projected/e2df4550-54e5-42d1-9257-077917d8d1b3-kube-api-access-pks6r\") pod \"e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr\" (UID: \"e2df4550-54e5-42d1-9257-077917d8d1b3\") " pod="openshift-ingress/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr" Sep 11 18:39:34.150773 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:34.150683 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr" Sep 11 18:39:34.433037 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:34.433004 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr"] Sep 11 18:39:34.434292 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:39:34.434259 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2df4550_54e5_42d1_9257_077917d8d1b3.slice/crio-2d6dc8e7afbe4638f01864dcd24b5358cacb55af6429e263c6b5adf21ab69ccc WatchSource:0}: Error finding container 2d6dc8e7afbe4638f01864dcd24b5358cacb55af6429e263c6b5adf21ab69ccc: Status 404 returned error can't find the container with id 2d6dc8e7afbe4638f01864dcd24b5358cacb55af6429e263c6b5adf21ab69ccc Sep 11 18:39:34.436736 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:34.436701 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:39:34.436812 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:34.436770 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:39:34.436812 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:34.436799 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:39:34.576641 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:34.576609 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr" event={"ID":"e2df4550-54e5-42d1-9257-077917d8d1b3","Type":"ContainerStarted","Data":"ac6ce281cf2e29072bae0323fbaa600f895ed5921528bc82b15592d3d7841dc0"} Sep 11 18:39:34.576844 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:34.576666 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr" event={"ID":"e2df4550-54e5-42d1-9257-077917d8d1b3","Type":"ContainerStarted","Data":"2d6dc8e7afbe4638f01864dcd24b5358cacb55af6429e263c6b5adf21ab69ccc"} Sep 11 18:39:34.597308 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:34.597237 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr" podStartSLOduration=1.597214913 podStartE2EDuration="1.597214913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:39:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:39:34.595279722 +0000 UTC m=+1456.092297488" watchObservedRunningTime="2026-09-11 18:39:34.597214913 +0000 UTC m=+1456.094232683" Sep 11 18:39:35.071089 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:35.071038 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-efa8b9/embedding-w0-efa8b9-kserve-6d8f694465-l5drh" Sep 11 18:39:35.071532 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:35.071112 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-efa8b9/embedding-w0-efa8b9-kserve-6d8f694465-l5drh" Sep 11 18:39:35.084227 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:35.084192 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-efa8b9/embedding-w0-efa8b9-kserve-6d8f694465-l5drh" Sep 11 18:39:35.151576 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:35.151539 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr" Sep 11 18:39:35.156724 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:35.156699 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr" Sep 11 18:39:35.584336 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:35.584301 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr" Sep 11 18:39:35.585587 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:35.585550 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr" Sep 11 18:39:35.600456 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:39:35.600430 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-efa8b9/embedding-w0-efa8b9-kserve-6d8f694465-l5drh" Sep 11 18:40:12.207544 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:12.207495 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ea789c-7996dc64b6-kjsmx"] Sep 11 18:40:12.212506 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:12.212460 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea789c-7996dc64b6-kjsmx" Sep 11 18:40:12.218342 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:12.218312 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-ea789c-serving-cert\"" Sep 11 18:40:12.226710 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:12.226678 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ea789c-7996dc64b6-kjsmx"] Sep 11 18:40:12.319461 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:12.319421 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/489ae584-3686-403e-bd3b-797e3f78868d-openshift-service-ca\") pod \"maas-api-e2e-api-b-ea789c-7996dc64b6-kjsmx\" (UID: \"489ae584-3686-403e-bd3b-797e3f78868d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea789c-7996dc64b6-kjsmx" Sep 11 18:40:12.319684 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:12.319492 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/489ae584-3686-403e-bd3b-797e3f78868d-metrics-certs\") pod \"maas-api-e2e-api-b-ea789c-7996dc64b6-kjsmx\" (UID: \"489ae584-3686-403e-bd3b-797e3f78868d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea789c-7996dc64b6-kjsmx" Sep 11 18:40:12.319684 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:12.319530 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cp8h\" (UniqueName: \"kubernetes.io/projected/489ae584-3686-403e-bd3b-797e3f78868d-kube-api-access-7cp8h\") pod \"maas-api-e2e-api-b-ea789c-7996dc64b6-kjsmx\" (UID: \"489ae584-3686-403e-bd3b-797e3f78868d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea789c-7996dc64b6-kjsmx" Sep 11 18:40:12.319812 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:12.319676 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/489ae584-3686-403e-bd3b-797e3f78868d-maas-api-tls\") pod \"maas-api-e2e-api-b-ea789c-7996dc64b6-kjsmx\" (UID: \"489ae584-3686-403e-bd3b-797e3f78868d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea789c-7996dc64b6-kjsmx" Sep 11 18:40:12.421367 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:12.421324 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/489ae584-3686-403e-bd3b-797e3f78868d-openshift-service-ca\") pod \"maas-api-e2e-api-b-ea789c-7996dc64b6-kjsmx\" (UID: \"489ae584-3686-403e-bd3b-797e3f78868d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea789c-7996dc64b6-kjsmx" Sep 11 18:40:12.421543 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:12.421390 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/489ae584-3686-403e-bd3b-797e3f78868d-metrics-certs\") pod \"maas-api-e2e-api-b-ea789c-7996dc64b6-kjsmx\" (UID: \"489ae584-3686-403e-bd3b-797e3f78868d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea789c-7996dc64b6-kjsmx" Sep 11 18:40:12.421543 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:12.421414 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cp8h\" (UniqueName: \"kubernetes.io/projected/489ae584-3686-403e-bd3b-797e3f78868d-kube-api-access-7cp8h\") pod \"maas-api-e2e-api-b-ea789c-7996dc64b6-kjsmx\" (UID: \"489ae584-3686-403e-bd3b-797e3f78868d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea789c-7996dc64b6-kjsmx" Sep 11 18:40:12.421543 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:12.421474 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/489ae584-3686-403e-bd3b-797e3f78868d-maas-api-tls\") pod \"maas-api-e2e-api-b-ea789c-7996dc64b6-kjsmx\" (UID: \"489ae584-3686-403e-bd3b-797e3f78868d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea789c-7996dc64b6-kjsmx" Sep 11 18:40:12.421654 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:40:12.421591 2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-ea789c-serving-cert: secret "maas-api-e2e-api-b-ea789c-serving-cert" not found Sep 11 18:40:12.421654 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:40:12.421650 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/489ae584-3686-403e-bd3b-797e3f78868d-maas-api-tls podName:489ae584-3686-403e-bd3b-797e3f78868d nodeName:}" failed. No retries permitted until 2026-09-11 18:40:12.921633319 +0000 UTC m=+1494.418651052 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/489ae584-3686-403e-bd3b-797e3f78868d-maas-api-tls") pod "maas-api-e2e-api-b-ea789c-7996dc64b6-kjsmx" (UID: "489ae584-3686-403e-bd3b-797e3f78868d") : secret "maas-api-e2e-api-b-ea789c-serving-cert" not found Sep 11 18:40:12.422289 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:12.422263 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/489ae584-3686-403e-bd3b-797e3f78868d-openshift-service-ca\") pod \"maas-api-e2e-api-b-ea789c-7996dc64b6-kjsmx\" (UID: \"489ae584-3686-403e-bd3b-797e3f78868d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea789c-7996dc64b6-kjsmx" Sep 11 18:40:12.424293 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:12.424271 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/489ae584-3686-403e-bd3b-797e3f78868d-metrics-certs\") pod \"maas-api-e2e-api-b-ea789c-7996dc64b6-kjsmx\" (UID: \"489ae584-3686-403e-bd3b-797e3f78868d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea789c-7996dc64b6-kjsmx" Sep 11 18:40:12.433327 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:12.433300 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cp8h\" (UniqueName: \"kubernetes.io/projected/489ae584-3686-403e-bd3b-797e3f78868d-kube-api-access-7cp8h\") pod \"maas-api-e2e-api-b-ea789c-7996dc64b6-kjsmx\" (UID: \"489ae584-3686-403e-bd3b-797e3f78868d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea789c-7996dc64b6-kjsmx" Sep 11 18:40:12.927182 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:12.927144 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/489ae584-3686-403e-bd3b-797e3f78868d-maas-api-tls\") pod \"maas-api-e2e-api-b-ea789c-7996dc64b6-kjsmx\" (UID: \"489ae584-3686-403e-bd3b-797e3f78868d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea789c-7996dc64b6-kjsmx" Sep 11 18:40:12.929847 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:12.929817 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/489ae584-3686-403e-bd3b-797e3f78868d-maas-api-tls\") pod \"maas-api-e2e-api-b-ea789c-7996dc64b6-kjsmx\" (UID: \"489ae584-3686-403e-bd3b-797e3f78868d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea789c-7996dc64b6-kjsmx" Sep 11 18:40:12.980913 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:12.980873 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ea789c-57f4777659-qxnc8"] Sep 11 18:40:12.992566 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:12.992533 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ea789c-57f4777659-qxnc8" Sep 11 18:40:12.996753 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:12.996724 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-ea789c-dockercfg-cwg5g\"" Sep 11 18:40:12.997260 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:12.997237 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-ea789c\"" Sep 11 18:40:13.000740 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:13.000703 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ea789c-57f4777659-qxnc8"] Sep 11 18:40:13.132080 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:13.130200 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwzgv\" (UniqueName: \"kubernetes.io/projected/0ae31332-0851-4f28-b6ad-096f486ce60a-kube-api-access-xwzgv\") pod \"payload-processing-e2e-api-b-ea789c-57f4777659-qxnc8\" (UID: \"0ae31332-0851-4f28-b6ad-096f486ce60a\") " pod="openshift-ingress/payload-processing-e2e-api-b-ea789c-57f4777659-qxnc8" Sep 11 18:40:13.132080 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:13.130300 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ae31332-0851-4f28-b6ad-096f486ce60a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ea789c-57f4777659-qxnc8\" (UID: \"0ae31332-0851-4f28-b6ad-096f486ce60a\") " pod="openshift-ingress/payload-processing-e2e-api-b-ea789c-57f4777659-qxnc8" Sep 11 18:40:13.136918 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:13.136885 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea789c-7996dc64b6-kjsmx" Sep 11 18:40:13.233359 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:13.232487 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwzgv\" (UniqueName: \"kubernetes.io/projected/0ae31332-0851-4f28-b6ad-096f486ce60a-kube-api-access-xwzgv\") pod \"payload-processing-e2e-api-b-ea789c-57f4777659-qxnc8\" (UID: \"0ae31332-0851-4f28-b6ad-096f486ce60a\") " pod="openshift-ingress/payload-processing-e2e-api-b-ea789c-57f4777659-qxnc8" Sep 11 18:40:13.233359 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:13.232584 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ae31332-0851-4f28-b6ad-096f486ce60a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ea789c-57f4777659-qxnc8\" (UID: \"0ae31332-0851-4f28-b6ad-096f486ce60a\") " pod="openshift-ingress/payload-processing-e2e-api-b-ea789c-57f4777659-qxnc8" Sep 11 18:40:13.233881 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:13.233560 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ae31332-0851-4f28-b6ad-096f486ce60a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ea789c-57f4777659-qxnc8\" (UID: \"0ae31332-0851-4f28-b6ad-096f486ce60a\") " pod="openshift-ingress/payload-processing-e2e-api-b-ea789c-57f4777659-qxnc8" Sep 11 18:40:13.241748 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:13.241716 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwzgv\" (UniqueName: \"kubernetes.io/projected/0ae31332-0851-4f28-b6ad-096f486ce60a-kube-api-access-xwzgv\") pod \"payload-processing-e2e-api-b-ea789c-57f4777659-qxnc8\" (UID: \"0ae31332-0851-4f28-b6ad-096f486ce60a\") " pod="openshift-ingress/payload-processing-e2e-api-b-ea789c-57f4777659-qxnc8" Sep 11 18:40:13.317859 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:13.317809 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ea789c-57f4777659-qxnc8" Sep 11 18:40:13.445662 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:13.445455 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ea789c-7996dc64b6-kjsmx"] Sep 11 18:40:13.446378 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:40:13.446350 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod489ae584_3686_403e_bd3b_797e3f78868d.slice/crio-805822da26d45ae0b82b574d46b0e4f7a9a97a7fc75b7319b995cd793940d21f WatchSource:0}: Error finding container 805822da26d45ae0b82b574d46b0e4f7a9a97a7fc75b7319b995cd793940d21f: Status 404 returned error can't find the container with id 805822da26d45ae0b82b574d46b0e4f7a9a97a7fc75b7319b995cd793940d21f Sep 11 18:40:13.448264 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:13.448199 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 18:40:13.609448 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:13.608962 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ea789c-57f4777659-qxnc8"] Sep 11 18:40:13.850974 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:13.850934 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ea789c-57f4777659-qxnc8" event={"ID":"0ae31332-0851-4f28-b6ad-096f486ce60a","Type":"ContainerStarted","Data":"92ba75bd6946fafc57ac4acd4bebfb028d789751ef8d84c4ea31c2f2ad543743"} Sep 11 18:40:13.851150 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:13.850999 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ea789c-57f4777659-qxnc8" event={"ID":"0ae31332-0851-4f28-b6ad-096f486ce60a","Type":"ContainerStarted","Data":"70749822f9ec717f2618f2a6ac934d3997d9ff61532f28f37ff17164857c4d58"} Sep 11 18:40:13.851150 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:13.851031 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-ea789c-57f4777659-qxnc8" Sep 11 18:40:13.852859 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:13.852835 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea789c-7996dc64b6-kjsmx" event={"ID":"489ae584-3686-403e-bd3b-797e3f78868d","Type":"ContainerStarted","Data":"805822da26d45ae0b82b574d46b0e4f7a9a97a7fc75b7319b995cd793940d21f"} Sep 11 18:40:13.867708 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:13.867582 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-ea789c-57f4777659-qxnc8" podStartSLOduration=1.867566775 podStartE2EDuration="1.867566775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:40:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:40:13.865906314 +0000 UTC m=+1495.362924080" watchObservedRunningTime="2026-09-11 18:40:13.867566775 +0000 UTC m=+1495.364584528" Sep 11 18:40:14.861749 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:14.861714 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea789c-7996dc64b6-kjsmx" event={"ID":"489ae584-3686-403e-bd3b-797e3f78868d","Type":"ContainerStarted","Data":"95a75f178b61b382930eccd73ba8188eeefd18772fd9693f747f6fed2df25bcd"} Sep 11 18:40:14.862178 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:14.862143 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea789c-7996dc64b6-kjsmx" Sep 11 18:40:14.880854 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:14.880804 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea789c-7996dc64b6-kjsmx" podStartSLOduration=2.507627315 podStartE2EDuration="2.880788676s" totalImagesPullingTime="373.161361ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:40:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:40:14.878235018 +0000 UTC m=+1496.375252771" watchObservedRunningTime="2026-09-11 18:40:14.880788676 +0000 UTC m=+1496.377806477" Sep 11 18:40:19.489775 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:19.489740 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dj524_60702ef3-98ef-463b-b877-3a00825353d8/console-operator/1.log" Sep 11 18:40:19.497511 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:19.497479 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dj524_60702ef3-98ef-463b-b877-3a00825353d8/console-operator/1.log" Sep 11 18:40:19.498730 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:19.498700 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4knd_42e5ddd8-a96c-45a1-adf9-caa47ee902db/ovn-acl-logging/0.log" Sep 11 18:40:19.506527 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:19.506501 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4knd_42e5ddd8-a96c-45a1-adf9-caa47ee902db/ovn-acl-logging/0.log" Sep 11 18:40:19.902066 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:19.902023 2561 generic.go:356] "Generic (PLEG): container finished" podID="e2df4550-54e5-42d1-9257-077917d8d1b3" containerID="ac6ce281cf2e29072bae0323fbaa600f895ed5921528bc82b15592d3d7841dc0" exitCode=0 Sep 11 18:40:19.902257 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:19.902103 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr" event={"ID":"e2df4550-54e5-42d1-9257-077917d8d1b3","Type":"ContainerDied","Data":"ac6ce281cf2e29072bae0323fbaa600f895ed5921528bc82b15592d3d7841dc0"} Sep 11 18:40:19.902576 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:19.902560 2561 scope.go:117] "RemoveContainer" containerID="ac6ce281cf2e29072bae0323fbaa600f895ed5921528bc82b15592d3d7841dc0" Sep 11 18:40:19.903880 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:19.903852 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:40:19.903937 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:19.903927 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:40:19.903974 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:19.903960 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:40:20.585102 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:20.585040 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr" Sep 11 18:40:20.914872 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:20.914833 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr" event={"ID":"e2df4550-54e5-42d1-9257-077917d8d1b3","Type":"ContainerStarted","Data":"69ece9ada8572acae2ec86e2c70953b7ff47ec0cf5f1a7fca85134fb450307b4"} Sep 11 18:40:21.151486 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:21.151444 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr" Sep 11 18:40:21.152892 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:21.152861 2561 patch_prober.go:28] interesting pod/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Sep 11 18:40:21.153035 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:21.152916 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr" podUID="e2df4550-54e5-42d1-9257-077917d8d1b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Sep 11 18:40:21.881614 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:21.881582 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea789c-7996dc64b6-kjsmx" Sep 11 18:40:22.151980 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:22.151887 2561 patch_prober.go:28] interesting pod/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Sep 11 18:40:22.151980 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:22.151956 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr" podUID="e2df4550-54e5-42d1-9257-077917d8d1b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Sep 11 18:40:23.151754 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:23.151718 2561 patch_prober.go:28] interesting pod/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Sep 11 18:40:23.152155 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:23.151772 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr" podUID="e2df4550-54e5-42d1-9257-077917d8d1b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Sep 11 18:40:24.151492 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:24.151454 2561 patch_prober.go:28] interesting pod/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Sep 11 18:40:24.151673 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:24.151511 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr" podUID="e2df4550-54e5-42d1-9257-077917d8d1b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Sep 11 18:40:24.864707 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:24.864675 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-ea789c-57f4777659-qxnc8" Sep 11 18:40:25.151756 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:25.151678 2561 patch_prober.go:28] interesting pod/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Sep 11 18:40:25.151979 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:25.151955 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr" podUID="e2df4550-54e5-42d1-9257-077917d8d1b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Sep 11 18:40:26.156350 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:26.156321 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr" Sep 11 18:40:26.156801 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:26.156784 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr" Sep 11 18:40:26.157484 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:26.157460 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr" Sep 11 18:40:44.076146 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:44.076102 2561 generic.go:356] "Generic (PLEG): container finished" podID="0059eaea-499d-499c-98d6-9547fc02722d" containerID="5f3653ee21cd4951b8d64ae6e2780bd1b6d3209aa53a3fc3f2a62995319f9f39" exitCode=137 Sep 11 18:40:44.076706 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:44.076174 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lsktr" event={"ID":"0059eaea-499d-499c-98d6-9547fc02722d","Type":"ContainerDied","Data":"5f3653ee21cd4951b8d64ae6e2780bd1b6d3209aa53a3fc3f2a62995319f9f39"} Sep 11 18:40:44.077118 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:44.077091 2561 scope.go:117] "RemoveContainer" containerID="5f3653ee21cd4951b8d64ae6e2780bd1b6d3209aa53a3fc3f2a62995319f9f39" Sep 11 18:40:44.468202 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:44.468165 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lsktr" Sep 11 18:40:45.093610 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:45.093571 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lsktr" event={"ID":"0059eaea-499d-499c-98d6-9547fc02722d","Type":"ContainerStarted","Data":"ce74c0a1160ab916eb52557874dc9bb5a904eb6127fd099f33a71743c580205a"} Sep 11 18:40:45.094112 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:45.093734 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lsktr" Sep 11 18:40:56.102566 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:40:56.102534 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lsktr" Sep 11 18:41:01.208751 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:01.208709 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr"] Sep 11 18:41:01.209231 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:01.208955 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr" podUID="e2df4550-54e5-42d1-9257-077917d8d1b3" containerName="istio-proxy" containerID="cri-o://69ece9ada8572acae2ec86e2c70953b7ff47ec0cf5f1a7fca85134fb450307b4" gracePeriod=30 Sep 11 18:41:06.480735 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:06.480669 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr" Sep 11 18:41:06.621015 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:06.620980 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2df4550-54e5-42d1-9257-077917d8d1b3-istio-token\") pod \"e2df4550-54e5-42d1-9257-077917d8d1b3\" (UID: \"e2df4550-54e5-42d1-9257-077917d8d1b3\") " Sep 11 18:41:06.621203 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:06.621040 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2df4550-54e5-42d1-9257-077917d8d1b3-istiod-ca-cert\") pod \"e2df4550-54e5-42d1-9257-077917d8d1b3\" (UID: \"e2df4550-54e5-42d1-9257-077917d8d1b3\") " Sep 11 18:41:06.621203 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:06.621122 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2df4550-54e5-42d1-9257-077917d8d1b3-workload-socket\") pod \"e2df4550-54e5-42d1-9257-077917d8d1b3\" (UID: \"e2df4550-54e5-42d1-9257-077917d8d1b3\") " Sep 11 18:41:06.621203 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:06.621154 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2df4550-54e5-42d1-9257-077917d8d1b3-istio-envoy\") pod \"e2df4550-54e5-42d1-9257-077917d8d1b3\" (UID: \"e2df4550-54e5-42d1-9257-077917d8d1b3\") " Sep 11 18:41:06.621203 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:06.621193 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2df4550-54e5-42d1-9257-077917d8d1b3-workload-certs\") pod \"e2df4550-54e5-42d1-9257-077917d8d1b3\" (UID: \"e2df4550-54e5-42d1-9257-077917d8d1b3\") " Sep 11 18:41:06.621444 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:06.621216 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2df4550-54e5-42d1-9257-077917d8d1b3-credential-socket\") pod \"e2df4550-54e5-42d1-9257-077917d8d1b3\" (UID: \"e2df4550-54e5-42d1-9257-077917d8d1b3\") " Sep 11 18:41:06.621444 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:06.621270 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2df4550-54e5-42d1-9257-077917d8d1b3-istio-data\") pod \"e2df4550-54e5-42d1-9257-077917d8d1b3\" (UID: \"e2df4550-54e5-42d1-9257-077917d8d1b3\") " Sep 11 18:41:06.621444 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:06.621322 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pks6r\" (UniqueName: \"kubernetes.io/projected/e2df4550-54e5-42d1-9257-077917d8d1b3-kube-api-access-pks6r\") pod \"e2df4550-54e5-42d1-9257-077917d8d1b3\" (UID: \"e2df4550-54e5-42d1-9257-077917d8d1b3\") " Sep 11 18:41:06.621444 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:06.621404 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2df4550-54e5-42d1-9257-077917d8d1b3-istio-podinfo\") pod \"e2df4550-54e5-42d1-9257-077917d8d1b3\" (UID: \"e2df4550-54e5-42d1-9257-077917d8d1b3\") " Sep 11 18:41:06.621660 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:06.621471 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2df4550-54e5-42d1-9257-077917d8d1b3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e2df4550-54e5-42d1-9257-077917d8d1b3" (UID: "e2df4550-54e5-42d1-9257-077917d8d1b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:41:06.621660 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:06.621491 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2df4550-54e5-42d1-9257-077917d8d1b3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e2df4550-54e5-42d1-9257-077917d8d1b3" (UID: "e2df4550-54e5-42d1-9257-077917d8d1b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:41:06.621660 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:06.621604 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2df4550-54e5-42d1-9257-077917d8d1b3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e2df4550-54e5-42d1-9257-077917d8d1b3" (UID: "e2df4550-54e5-42d1-9257-077917d8d1b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:41:06.621660 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:06.621638 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2df4550-54e5-42d1-9257-077917d8d1b3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e2df4550-54e5-42d1-9257-077917d8d1b3" (UID: "e2df4550-54e5-42d1-9257-077917d8d1b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:41:06.621884 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:06.621659 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2df4550-54e5-42d1-9257-077917d8d1b3-istio-data" (OuterVolumeSpecName: "istio-data") pod "e2df4550-54e5-42d1-9257-077917d8d1b3" (UID: "e2df4550-54e5-42d1-9257-077917d8d1b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:41:06.621884 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:06.621832 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2df4550-54e5-42d1-9257-077917d8d1b3-istiod-ca-cert\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:41:06.621884 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:06.621874 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2df4550-54e5-42d1-9257-077917d8d1b3-workload-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:41:06.622045 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:06.621890 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2df4550-54e5-42d1-9257-077917d8d1b3-workload-certs\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:41:06.622045 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:06.621903 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2df4550-54e5-42d1-9257-077917d8d1b3-credential-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:41:06.622045 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:06.621916 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2df4550-54e5-42d1-9257-077917d8d1b3-istio-data\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:41:06.623631 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:06.623592 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2df4550-54e5-42d1-9257-077917d8d1b3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e2df4550-54e5-42d1-9257-077917d8d1b3" (UID: "e2df4550-54e5-42d1-9257-077917d8d1b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 18:41:06.623631 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:06.623596 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2df4550-54e5-42d1-9257-077917d8d1b3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e2df4550-54e5-42d1-9257-077917d8d1b3" (UID: "e2df4550-54e5-42d1-9257-077917d8d1b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:41:06.623789 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:06.623678 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2df4550-54e5-42d1-9257-077917d8d1b3-kube-api-access-pks6r" (OuterVolumeSpecName: "kube-api-access-pks6r") pod "e2df4550-54e5-42d1-9257-077917d8d1b3" (UID: "e2df4550-54e5-42d1-9257-077917d8d1b3"). InnerVolumeSpecName "kube-api-access-pks6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:41:06.623925 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:06.623903 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2df4550-54e5-42d1-9257-077917d8d1b3-istio-token" (OuterVolumeSpecName: "istio-token") pod "e2df4550-54e5-42d1-9257-077917d8d1b3" (UID: "e2df4550-54e5-42d1-9257-077917d8d1b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:41:06.723003 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:06.722961 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pks6r\" (UniqueName: \"kubernetes.io/projected/e2df4550-54e5-42d1-9257-077917d8d1b3-kube-api-access-pks6r\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:41:06.723003 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:06.722996 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2df4550-54e5-42d1-9257-077917d8d1b3-istio-podinfo\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:41:06.723003 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:06.723010 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2df4550-54e5-42d1-9257-077917d8d1b3-istio-token\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:41:06.723288 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:06.723025 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2df4550-54e5-42d1-9257-077917d8d1b3-istio-envoy\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:41:07.258472 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:07.258432 2561 generic.go:356] "Generic (PLEG): container finished" podID="e2df4550-54e5-42d1-9257-077917d8d1b3" containerID="69ece9ada8572acae2ec86e2c70953b7ff47ec0cf5f1a7fca85134fb450307b4" exitCode=0 Sep 11 18:41:07.258643 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:07.258489 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr" Sep 11 18:41:07.258643 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:07.258514 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr" event={"ID":"e2df4550-54e5-42d1-9257-077917d8d1b3","Type":"ContainerDied","Data":"69ece9ada8572acae2ec86e2c70953b7ff47ec0cf5f1a7fca85134fb450307b4"} Sep 11 18:41:07.258643 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:07.258571 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr" event={"ID":"e2df4550-54e5-42d1-9257-077917d8d1b3","Type":"ContainerDied","Data":"2d6dc8e7afbe4638f01864dcd24b5358cacb55af6429e263c6b5adf21ab69ccc"} Sep 11 18:41:07.258643 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:07.258593 2561 scope.go:117] "RemoveContainer" containerID="69ece9ada8572acae2ec86e2c70953b7ff47ec0cf5f1a7fca85134fb450307b4" Sep 11 18:41:07.269268 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:07.269243 2561 scope.go:117] "RemoveContainer" containerID="ac6ce281cf2e29072bae0323fbaa600f895ed5921528bc82b15592d3d7841dc0" Sep 11 18:41:07.279029 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:07.279007 2561 scope.go:117] "RemoveContainer" containerID="69ece9ada8572acae2ec86e2c70953b7ff47ec0cf5f1a7fca85134fb450307b4" Sep 11 18:41:07.279380 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:41:07.279358 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69ece9ada8572acae2ec86e2c70953b7ff47ec0cf5f1a7fca85134fb450307b4\": container with ID starting with 69ece9ada8572acae2ec86e2c70953b7ff47ec0cf5f1a7fca85134fb450307b4 not found: ID does not exist" containerID="69ece9ada8572acae2ec86e2c70953b7ff47ec0cf5f1a7fca85134fb450307b4" Sep 11 18:41:07.279478 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:07.279389 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69ece9ada8572acae2ec86e2c70953b7ff47ec0cf5f1a7fca85134fb450307b4"} err="failed to get container status \"69ece9ada8572acae2ec86e2c70953b7ff47ec0cf5f1a7fca85134fb450307b4\": rpc error: code = NotFound desc = could not find container \"69ece9ada8572acae2ec86e2c70953b7ff47ec0cf5f1a7fca85134fb450307b4\": container with ID starting with 69ece9ada8572acae2ec86e2c70953b7ff47ec0cf5f1a7fca85134fb450307b4 not found: ID does not exist" Sep 11 18:41:07.279478 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:07.279408 2561 scope.go:117] "RemoveContainer" containerID="ac6ce281cf2e29072bae0323fbaa600f895ed5921528bc82b15592d3d7841dc0" Sep 11 18:41:07.279687 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:41:07.279664 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac6ce281cf2e29072bae0323fbaa600f895ed5921528bc82b15592d3d7841dc0\": container with ID starting with ac6ce281cf2e29072bae0323fbaa600f895ed5921528bc82b15592d3d7841dc0 not found: ID does not exist" containerID="ac6ce281cf2e29072bae0323fbaa600f895ed5921528bc82b15592d3d7841dc0" Sep 11 18:41:07.279744 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:07.279695 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac6ce281cf2e29072bae0323fbaa600f895ed5921528bc82b15592d3d7841dc0"} err="failed to get container status \"ac6ce281cf2e29072bae0323fbaa600f895ed5921528bc82b15592d3d7841dc0\": rpc error: code = NotFound desc = could not find container \"ac6ce281cf2e29072bae0323fbaa600f895ed5921528bc82b15592d3d7841dc0\": container with ID starting with ac6ce281cf2e29072bae0323fbaa600f895ed5921528bc82b15592d3d7841dc0 not found: ID does not exist" Sep 11 18:41:07.279970 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:07.279948 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr"] Sep 11 18:41:07.284785 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:07.284760 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-c1409b-openshift-default-65d995d7c4-d8cjr"] Sep 11 18:41:09.148606 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:09.148570 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2df4550-54e5-42d1-9257-077917d8d1b3" path="/var/lib/kubelet/pods/e2df4550-54e5-42d1-9257-077917d8d1b3/volumes" Sep 11 18:41:17.333926 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:17.333889 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ea789c-7996dc64b6-kjsmx"] Sep 11 18:41:17.334522 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:17.334152 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea789c-7996dc64b6-kjsmx" podUID="489ae584-3686-403e-bd3b-797e3f78868d" containerName="maas-api" containerID="cri-o://95a75f178b61b382930eccd73ba8188eeefd18772fd9693f747f6fed2df25bcd" gracePeriod=30 Sep 11 18:41:17.393315 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:17.393277 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ea789c-57f4777659-qxnc8"] Sep 11 18:41:17.393788 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:17.393758 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-ea789c-57f4777659-qxnc8" podUID="0ae31332-0851-4f28-b6ad-096f486ce60a" containerName="payload-processing" containerID="cri-o://92ba75bd6946fafc57ac4acd4bebfb028d789751ef8d84c4ea31c2f2ad543743" gracePeriod=30 Sep 11 18:41:17.657819 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:17.657788 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea789c-7996dc64b6-kjsmx" Sep 11 18:41:17.707027 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:17.706986 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ea789c-57f4777659-qxnc8" Sep 11 18:41:17.742448 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:17.742409 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/489ae584-3686-403e-bd3b-797e3f78868d-openshift-service-ca\") pod \"489ae584-3686-403e-bd3b-797e3f78868d\" (UID: \"489ae584-3686-403e-bd3b-797e3f78868d\") " Sep 11 18:41:17.742448 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:17.742448 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/489ae584-3686-403e-bd3b-797e3f78868d-metrics-certs\") pod \"489ae584-3686-403e-bd3b-797e3f78868d\" (UID: \"489ae584-3686-403e-bd3b-797e3f78868d\") " Sep 11 18:41:17.742683 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:17.742544 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7cp8h\" (UniqueName: \"kubernetes.io/projected/489ae584-3686-403e-bd3b-797e3f78868d-kube-api-access-7cp8h\") pod \"489ae584-3686-403e-bd3b-797e3f78868d\" (UID: \"489ae584-3686-403e-bd3b-797e3f78868d\") " Sep 11 18:41:17.742683 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:17.742575 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/489ae584-3686-403e-bd3b-797e3f78868d-maas-api-tls\") pod \"489ae584-3686-403e-bd3b-797e3f78868d\" (UID: \"489ae584-3686-403e-bd3b-797e3f78868d\") " Sep 11 18:41:17.742853 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:17.742822 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/489ae584-3686-403e-bd3b-797e3f78868d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "489ae584-3686-403e-bd3b-797e3f78868d" (UID: "489ae584-3686-403e-bd3b-797e3f78868d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:41:17.743032 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:17.743011 2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/489ae584-3686-403e-bd3b-797e3f78868d-openshift-service-ca\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:41:17.745047 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:17.745009 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/489ae584-3686-403e-bd3b-797e3f78868d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "489ae584-3686-403e-bd3b-797e3f78868d" (UID: "489ae584-3686-403e-bd3b-797e3f78868d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:41:17.745183 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:17.745098 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/489ae584-3686-403e-bd3b-797e3f78868d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "489ae584-3686-403e-bd3b-797e3f78868d" (UID: "489ae584-3686-403e-bd3b-797e3f78868d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:41:17.745183 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:17.745143 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/489ae584-3686-403e-bd3b-797e3f78868d-kube-api-access-7cp8h" (OuterVolumeSpecName: "kube-api-access-7cp8h") pod "489ae584-3686-403e-bd3b-797e3f78868d" (UID: "489ae584-3686-403e-bd3b-797e3f78868d"). InnerVolumeSpecName "kube-api-access-7cp8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:41:17.843905 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:17.843860 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xwzgv\" (UniqueName: \"kubernetes.io/projected/0ae31332-0851-4f28-b6ad-096f486ce60a-kube-api-access-xwzgv\") pod \"0ae31332-0851-4f28-b6ad-096f486ce60a\" (UID: \"0ae31332-0851-4f28-b6ad-096f486ce60a\") " Sep 11 18:41:17.843905 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:17.843909 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ae31332-0851-4f28-b6ad-096f486ce60a-plugins-config-volume\") pod \"0ae31332-0851-4f28-b6ad-096f486ce60a\" (UID: \"0ae31332-0851-4f28-b6ad-096f486ce60a\") " Sep 11 18:41:17.844194 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:17.844182 2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/489ae584-3686-403e-bd3b-797e3f78868d-metrics-certs\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:41:17.844236 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:17.844196 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cp8h\" (UniqueName: \"kubernetes.io/projected/489ae584-3686-403e-bd3b-797e3f78868d-kube-api-access-7cp8h\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:41:17.844236 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:17.844206 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/489ae584-3686-403e-bd3b-797e3f78868d-maas-api-tls\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:41:17.844343 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:17.844314 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ae31332-0851-4f28-b6ad-096f486ce60a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0ae31332-0851-4f28-b6ad-096f486ce60a" (UID: "0ae31332-0851-4f28-b6ad-096f486ce60a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:41:17.846008 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:17.845986 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ae31332-0851-4f28-b6ad-096f486ce60a-kube-api-access-xwzgv" (OuterVolumeSpecName: "kube-api-access-xwzgv") pod "0ae31332-0851-4f28-b6ad-096f486ce60a" (UID: "0ae31332-0851-4f28-b6ad-096f486ce60a"). InnerVolumeSpecName "kube-api-access-xwzgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:41:17.945532 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:17.945490 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwzgv\" (UniqueName: \"kubernetes.io/projected/0ae31332-0851-4f28-b6ad-096f486ce60a-kube-api-access-xwzgv\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:41:17.945532 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:17.945525 2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ae31332-0851-4f28-b6ad-096f486ce60a-plugins-config-volume\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:41:18.339977 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:18.339890 2561 generic.go:356] "Generic (PLEG): container finished" podID="0ae31332-0851-4f28-b6ad-096f486ce60a" containerID="92ba75bd6946fafc57ac4acd4bebfb028d789751ef8d84c4ea31c2f2ad543743" exitCode=0 Sep 11 18:41:18.339977 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:18.339929 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ea789c-57f4777659-qxnc8" Sep 11 18:41:18.340484 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:18.339971 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ea789c-57f4777659-qxnc8" event={"ID":"0ae31332-0851-4f28-b6ad-096f486ce60a","Type":"ContainerDied","Data":"92ba75bd6946fafc57ac4acd4bebfb028d789751ef8d84c4ea31c2f2ad543743"} Sep 11 18:41:18.340484 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:18.340018 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ea789c-57f4777659-qxnc8" event={"ID":"0ae31332-0851-4f28-b6ad-096f486ce60a","Type":"ContainerDied","Data":"70749822f9ec717f2618f2a6ac934d3997d9ff61532f28f37ff17164857c4d58"} Sep 11 18:41:18.340484 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:18.340040 2561 scope.go:117] "RemoveContainer" containerID="92ba75bd6946fafc57ac4acd4bebfb028d789751ef8d84c4ea31c2f2ad543743" Sep 11 18:41:18.341757 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:18.341728 2561 generic.go:356] "Generic (PLEG): container finished" podID="489ae584-3686-403e-bd3b-797e3f78868d" containerID="95a75f178b61b382930eccd73ba8188eeefd18772fd9693f747f6fed2df25bcd" exitCode=0 Sep 11 18:41:18.341889 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:18.341782 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea789c-7996dc64b6-kjsmx" Sep 11 18:41:18.341889 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:18.341819 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea789c-7996dc64b6-kjsmx" event={"ID":"489ae584-3686-403e-bd3b-797e3f78868d","Type":"ContainerDied","Data":"95a75f178b61b382930eccd73ba8188eeefd18772fd9693f747f6fed2df25bcd"} Sep 11 18:41:18.341889 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:18.341871 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea789c-7996dc64b6-kjsmx" event={"ID":"489ae584-3686-403e-bd3b-797e3f78868d","Type":"ContainerDied","Data":"805822da26d45ae0b82b574d46b0e4f7a9a97a7fc75b7319b995cd793940d21f"} Sep 11 18:41:18.362829 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:18.362808 2561 scope.go:117] "RemoveContainer" containerID="92ba75bd6946fafc57ac4acd4bebfb028d789751ef8d84c4ea31c2f2ad543743" Sep 11 18:41:18.363186 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:41:18.363165 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92ba75bd6946fafc57ac4acd4bebfb028d789751ef8d84c4ea31c2f2ad543743\": container with ID starting with 92ba75bd6946fafc57ac4acd4bebfb028d789751ef8d84c4ea31c2f2ad543743 not found: ID does not exist" containerID="92ba75bd6946fafc57ac4acd4bebfb028d789751ef8d84c4ea31c2f2ad543743" Sep 11 18:41:18.363249 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:18.363196 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92ba75bd6946fafc57ac4acd4bebfb028d789751ef8d84c4ea31c2f2ad543743"} err="failed to get container status \"92ba75bd6946fafc57ac4acd4bebfb028d789751ef8d84c4ea31c2f2ad543743\": rpc error: code = NotFound desc = could not find container \"92ba75bd6946fafc57ac4acd4bebfb028d789751ef8d84c4ea31c2f2ad543743\": container with ID starting with 92ba75bd6946fafc57ac4acd4bebfb028d789751ef8d84c4ea31c2f2ad543743 not found: ID does not exist" Sep 11 18:41:18.363249 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:18.363213 2561 scope.go:117] "RemoveContainer" containerID="95a75f178b61b382930eccd73ba8188eeefd18772fd9693f747f6fed2df25bcd" Sep 11 18:41:18.372432 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:18.372411 2561 scope.go:117] "RemoveContainer" containerID="95a75f178b61b382930eccd73ba8188eeefd18772fd9693f747f6fed2df25bcd" Sep 11 18:41:18.372735 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:41:18.372714 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95a75f178b61b382930eccd73ba8188eeefd18772fd9693f747f6fed2df25bcd\": container with ID starting with 95a75f178b61b382930eccd73ba8188eeefd18772fd9693f747f6fed2df25bcd not found: ID does not exist" containerID="95a75f178b61b382930eccd73ba8188eeefd18772fd9693f747f6fed2df25bcd" Sep 11 18:41:18.372785 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:18.372743 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95a75f178b61b382930eccd73ba8188eeefd18772fd9693f747f6fed2df25bcd"} err="failed to get container status \"95a75f178b61b382930eccd73ba8188eeefd18772fd9693f747f6fed2df25bcd\": rpc error: code = NotFound desc = could not find container \"95a75f178b61b382930eccd73ba8188eeefd18772fd9693f747f6fed2df25bcd\": container with ID starting with 95a75f178b61b382930eccd73ba8188eeefd18772fd9693f747f6fed2df25bcd not found: ID does not exist" Sep 11 18:41:18.378572 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:18.378545 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ea789c-7996dc64b6-kjsmx"] Sep 11 18:41:18.383407 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:18.383359 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ea789c-7996dc64b6-kjsmx"] Sep 11 18:41:18.395547 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:18.395517 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ea789c-57f4777659-qxnc8"] Sep 11 18:41:18.399365 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:18.399338 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ea789c-57f4777659-qxnc8"] Sep 11 18:41:19.141702 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:19.141667 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ae31332-0851-4f28-b6ad-096f486ce60a" path="/var/lib/kubelet/pods/0ae31332-0851-4f28-b6ad-096f486ce60a/volumes" Sep 11 18:41:19.142144 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:19.142130 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="489ae584-3686-403e-bd3b-797e3f78868d" path="/var/lib/kubelet/pods/489ae584-3686-403e-bd3b-797e3f78868d/volumes" Sep 11 18:41:19.710443 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:19.710407 2561 scope.go:117] "RemoveContainer" containerID="22bc778d232453f5254340f465a641fc823e8a5ddf5f448b7344072dd84c91e5" Sep 11 18:41:19.720614 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:19.720585 2561 scope.go:117] "RemoveContainer" containerID="b0b7b41054e8e6da8b22209803065ee1528600c04011c9b30721a9a979b16713" Sep 11 18:41:35.280676 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:35.280636 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-869176-6c9cfbf7c8-bktwt"] Sep 11 18:41:35.281149 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:35.280896 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-869176-6c9cfbf7c8-bktwt" podUID="1b4839f5-19a5-4f78-88d5-99d06b629f30" containerName="payload-pre-processing" containerID="cri-o://dc551d9cdc0d44fe40bb52c8b93676cf8bd79e9f1a444cec85500fdd2cbc45f8" gracePeriod=30 Sep 11 18:41:35.494495 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:35.494299 2561 generic.go:356] "Generic (PLEG): container finished" podID="1b4839f5-19a5-4f78-88d5-99d06b629f30" containerID="dc551d9cdc0d44fe40bb52c8b93676cf8bd79e9f1a444cec85500fdd2cbc45f8" exitCode=0 Sep 11 18:41:35.494495 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:35.494407 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-869176-6c9cfbf7c8-bktwt" event={"ID":"1b4839f5-19a5-4f78-88d5-99d06b629f30","Type":"ContainerDied","Data":"dc551d9cdc0d44fe40bb52c8b93676cf8bd79e9f1a444cec85500fdd2cbc45f8"} Sep 11 18:41:35.575529 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:35.575502 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-869176-6c9cfbf7c8-bktwt" Sep 11 18:41:35.634932 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:35.634880 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b4839f5-19a5-4f78-88d5-99d06b629f30-plugins-config-volume\") pod \"1b4839f5-19a5-4f78-88d5-99d06b629f30\" (UID: \"1b4839f5-19a5-4f78-88d5-99d06b629f30\") " Sep 11 18:41:35.635164 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:35.635143 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kzkrp\" (UniqueName: \"kubernetes.io/projected/1b4839f5-19a5-4f78-88d5-99d06b629f30-kube-api-access-kzkrp\") pod \"1b4839f5-19a5-4f78-88d5-99d06b629f30\" (UID: \"1b4839f5-19a5-4f78-88d5-99d06b629f30\") " Sep 11 18:41:35.635455 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:35.635421 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b4839f5-19a5-4f78-88d5-99d06b629f30-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1b4839f5-19a5-4f78-88d5-99d06b629f30" (UID: "1b4839f5-19a5-4f78-88d5-99d06b629f30"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:41:35.635567 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:35.635551 2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b4839f5-19a5-4f78-88d5-99d06b629f30-plugins-config-volume\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:41:35.637517 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:35.637490 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b4839f5-19a5-4f78-88d5-99d06b629f30-kube-api-access-kzkrp" (OuterVolumeSpecName: "kube-api-access-kzkrp") pod "1b4839f5-19a5-4f78-88d5-99d06b629f30" (UID: "1b4839f5-19a5-4f78-88d5-99d06b629f30"). InnerVolumeSpecName "kube-api-access-kzkrp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:41:35.736879 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:35.736842 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzkrp\" (UniqueName: \"kubernetes.io/projected/1b4839f5-19a5-4f78-88d5-99d06b629f30-kube-api-access-kzkrp\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:41:36.503406 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:36.503377 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-869176-6c9cfbf7c8-bktwt" Sep 11 18:41:36.503936 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:36.503402 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-869176-6c9cfbf7c8-bktwt" event={"ID":"1b4839f5-19a5-4f78-88d5-99d06b629f30","Type":"ContainerDied","Data":"4d2c001c11981da3df615eb64bb75ea4012efccbaa51eb7080eb5f34e3c0de27"} Sep 11 18:41:36.503936 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:36.503469 2561 scope.go:117] "RemoveContainer" containerID="dc551d9cdc0d44fe40bb52c8b93676cf8bd79e9f1a444cec85500fdd2cbc45f8" Sep 11 18:41:36.530690 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:36.530655 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-869176-6c9cfbf7c8-bktwt"] Sep 11 18:41:36.535573 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:36.535545 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-869176-6c9cfbf7c8-bktwt"] Sep 11 18:41:36.849191 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:36.849098 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-efa8b9/embedding-w0-efa8b9-kserve-6d8f694465-l5drh"] Sep 11 18:41:37.139731 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:37.139649 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b4839f5-19a5-4f78-88d5-99d06b629f30" path="/var/lib/kubelet/pods/1b4839f5-19a5-4f78-88d5-99d06b629f30/volumes" Sep 11 18:41:43.371598 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:43.371561 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t"] Sep 11 18:41:43.371981 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:43.371874 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t" podUID="3ecbed32-91f0-44b6-aa50-2ed6f61bdc98" containerName="istio-proxy" containerID="cri-o://bc05afa511da53b8afe052665faabcd120ca9c792dff1261494428dab65c9623" gracePeriod=30 Sep 11 18:41:45.261149 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:45.261112 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-efa8b9-29819190-vjtwg"] Sep 11 18:41:45.264127 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:45.264100 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-efa8b9-29819190-vjtwg"] Sep 11 18:41:46.399351 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:46.399312 2561 patch_prober.go:28] interesting pod/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 18:41:46.399748 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:46.399374 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t" podUID="3ecbed32-91f0-44b6-aa50-2ed6f61bdc98" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 18:41:47.140262 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:47.140230 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="079c554e-e599-4ba7-a259-0a5e8cafad30" path="/var/lib/kubelet/pods/079c554e-e599-4ba7-a259-0a5e8cafad30/volumes" Sep 11 18:41:48.595120 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:48.595082 2561 generic.go:356] "Generic (PLEG): container finished" podID="3ecbed32-91f0-44b6-aa50-2ed6f61bdc98" containerID="bc05afa511da53b8afe052665faabcd120ca9c792dff1261494428dab65c9623" exitCode=0 Sep 11 18:41:48.595120 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:48.595098 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t" event={"ID":"3ecbed32-91f0-44b6-aa50-2ed6f61bdc98","Type":"ContainerDied","Data":"bc05afa511da53b8afe052665faabcd120ca9c792dff1261494428dab65c9623"} Sep 11 18:41:48.595557 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:48.595169 2561 scope.go:117] "RemoveContainer" containerID="ed8de445da68f2b789e2f88fe1bcf7caf6e5e4be6bf9129dd526e5ce93399c3e" Sep 11 18:41:48.648159 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:48.648095 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t" Sep 11 18:41:48.779395 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:48.779352 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ecbed32-91f0-44b6-aa50-2ed6f61bdc98-istiod-ca-cert\") pod \"3ecbed32-91f0-44b6-aa50-2ed6f61bdc98\" (UID: \"3ecbed32-91f0-44b6-aa50-2ed6f61bdc98\") " Sep 11 18:41:48.779563 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:48.779403 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ecbed32-91f0-44b6-aa50-2ed6f61bdc98-workload-certs\") pod \"3ecbed32-91f0-44b6-aa50-2ed6f61bdc98\" (UID: \"3ecbed32-91f0-44b6-aa50-2ed6f61bdc98\") " Sep 11 18:41:48.779563 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:48.779437 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ecbed32-91f0-44b6-aa50-2ed6f61bdc98-istio-envoy\") pod \"3ecbed32-91f0-44b6-aa50-2ed6f61bdc98\" (UID: \"3ecbed32-91f0-44b6-aa50-2ed6f61bdc98\") " Sep 11 18:41:48.779563 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:48.779549 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-94sd2\" (UniqueName: \"kubernetes.io/projected/3ecbed32-91f0-44b6-aa50-2ed6f61bdc98-kube-api-access-94sd2\") pod \"3ecbed32-91f0-44b6-aa50-2ed6f61bdc98\" (UID: \"3ecbed32-91f0-44b6-aa50-2ed6f61bdc98\") " Sep 11 18:41:48.779680 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:48.779592 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ecbed32-91f0-44b6-aa50-2ed6f61bdc98-credential-socket\") pod \"3ecbed32-91f0-44b6-aa50-2ed6f61bdc98\" (UID: \"3ecbed32-91f0-44b6-aa50-2ed6f61bdc98\") " Sep 11 18:41:48.779680 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:48.779660 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ecbed32-91f0-44b6-aa50-2ed6f61bdc98-istio-data\") pod \"3ecbed32-91f0-44b6-aa50-2ed6f61bdc98\" (UID: \"3ecbed32-91f0-44b6-aa50-2ed6f61bdc98\") " Sep 11 18:41:48.779766 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:48.779704 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ecbed32-91f0-44b6-aa50-2ed6f61bdc98-istio-token\") pod \"3ecbed32-91f0-44b6-aa50-2ed6f61bdc98\" (UID: \"3ecbed32-91f0-44b6-aa50-2ed6f61bdc98\") " Sep 11 18:41:48.779766 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:48.779714 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ecbed32-91f0-44b6-aa50-2ed6f61bdc98-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3ecbed32-91f0-44b6-aa50-2ed6f61bdc98" (UID: "3ecbed32-91f0-44b6-aa50-2ed6f61bdc98"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:41:48.779766 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:48.779755 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ecbed32-91f0-44b6-aa50-2ed6f61bdc98-istio-podinfo\") pod \"3ecbed32-91f0-44b6-aa50-2ed6f61bdc98\" (UID: \"3ecbed32-91f0-44b6-aa50-2ed6f61bdc98\") " Sep 11 18:41:48.779927 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:48.779792 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ecbed32-91f0-44b6-aa50-2ed6f61bdc98-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3ecbed32-91f0-44b6-aa50-2ed6f61bdc98" (UID: "3ecbed32-91f0-44b6-aa50-2ed6f61bdc98"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:41:48.779927 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:48.779807 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ecbed32-91f0-44b6-aa50-2ed6f61bdc98-workload-socket\") pod \"3ecbed32-91f0-44b6-aa50-2ed6f61bdc98\" (UID: \"3ecbed32-91f0-44b6-aa50-2ed6f61bdc98\") " Sep 11 18:41:48.779927 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:48.779861 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ecbed32-91f0-44b6-aa50-2ed6f61bdc98-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3ecbed32-91f0-44b6-aa50-2ed6f61bdc98" (UID: "3ecbed32-91f0-44b6-aa50-2ed6f61bdc98"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:41:48.780105 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:48.779985 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ecbed32-91f0-44b6-aa50-2ed6f61bdc98-istio-data" (OuterVolumeSpecName: "istio-data") pod "3ecbed32-91f0-44b6-aa50-2ed6f61bdc98" (UID: "3ecbed32-91f0-44b6-aa50-2ed6f61bdc98"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:41:48.780173 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:48.780101 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ecbed32-91f0-44b6-aa50-2ed6f61bdc98-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3ecbed32-91f0-44b6-aa50-2ed6f61bdc98" (UID: "3ecbed32-91f0-44b6-aa50-2ed6f61bdc98"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:41:48.780282 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:48.780263 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ecbed32-91f0-44b6-aa50-2ed6f61bdc98-credential-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:41:48.780346 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:48.780288 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ecbed32-91f0-44b6-aa50-2ed6f61bdc98-istio-data\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:41:48.780346 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:48.780304 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ecbed32-91f0-44b6-aa50-2ed6f61bdc98-workload-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:41:48.780346 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:48.780317 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ecbed32-91f0-44b6-aa50-2ed6f61bdc98-istiod-ca-cert\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:41:48.780346 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:48.780331 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ecbed32-91f0-44b6-aa50-2ed6f61bdc98-workload-certs\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:41:48.781948 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:48.781921 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ecbed32-91f0-44b6-aa50-2ed6f61bdc98-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3ecbed32-91f0-44b6-aa50-2ed6f61bdc98" (UID: "3ecbed32-91f0-44b6-aa50-2ed6f61bdc98"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 18:41:48.782036 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:48.781963 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ecbed32-91f0-44b6-aa50-2ed6f61bdc98-istio-token" (OuterVolumeSpecName: "istio-token") pod "3ecbed32-91f0-44b6-aa50-2ed6f61bdc98" (UID: "3ecbed32-91f0-44b6-aa50-2ed6f61bdc98"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:41:48.782260 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:48.782233 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ecbed32-91f0-44b6-aa50-2ed6f61bdc98-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3ecbed32-91f0-44b6-aa50-2ed6f61bdc98" (UID: "3ecbed32-91f0-44b6-aa50-2ed6f61bdc98"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:41:48.782260 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:48.782247 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ecbed32-91f0-44b6-aa50-2ed6f61bdc98-kube-api-access-94sd2" (OuterVolumeSpecName: "kube-api-access-94sd2") pod "3ecbed32-91f0-44b6-aa50-2ed6f61bdc98" (UID: "3ecbed32-91f0-44b6-aa50-2ed6f61bdc98"). InnerVolumeSpecName "kube-api-access-94sd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:41:48.881485 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:48.881446 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ecbed32-91f0-44b6-aa50-2ed6f61bdc98-istio-token\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:41:48.881485 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:48.881480 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ecbed32-91f0-44b6-aa50-2ed6f61bdc98-istio-podinfo\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:41:48.881485 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:48.881489 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ecbed32-91f0-44b6-aa50-2ed6f61bdc98-istio-envoy\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:41:48.881724 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:48.881498 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94sd2\" (UniqueName: \"kubernetes.io/projected/3ecbed32-91f0-44b6-aa50-2ed6f61bdc98-kube-api-access-94sd2\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:41:49.603391 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:49.603353 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t" Sep 11 18:41:49.603817 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:49.603387 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t" event={"ID":"3ecbed32-91f0-44b6-aa50-2ed6f61bdc98","Type":"ContainerDied","Data":"c2364f8ec15947ef3fd966c785a6dc29a895461287cfda3b0b4323a6bd3ddd78"} Sep 11 18:41:49.603817 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:49.603446 2561 scope.go:117] "RemoveContainer" containerID="bc05afa511da53b8afe052665faabcd120ca9c792dff1261494428dab65c9623" Sep 11 18:41:49.623526 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:49.623489 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t"] Sep 11 18:41:49.628401 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:49.628369 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-869176-openshift-default-69ff58987b-9fx2t"] Sep 11 18:41:51.139810 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:51.139773 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ecbed32-91f0-44b6-aa50-2ed6f61bdc98" path="/var/lib/kubelet/pods/3ecbed32-91f0-44b6-aa50-2ed6f61bdc98/volumes" Sep 11 18:41:51.870071 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:51.870020 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-efa8b9/embedding-w0-efa8b9-kserve-6d8f694465-l5drh" podUID="5e8a1657-b905-4add-84b2-5549f720a1d2" containerName="main" containerID="cri-o://f73ee44416f4ffcbe319e6d08535687099e4d7ecdec454822198132e4a3a042f" gracePeriod=45 Sep 11 18:41:52.137782 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:52.137707 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-efa8b9/embedding-w0-efa8b9-kserve-6d8f694465-l5drh" Sep 11 18:41:52.163934 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:52.163900 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm"] Sep 11 18:41:52.164644 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:52.164196 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm" podUID="891f8576-360b-4004-b612-2169d1323649" containerName="istio-proxy" containerID="cri-o://9db63a3867e2f67cc94e3282540a39d880f3cdca10d549c085bd50cceb730fdc" gracePeriod=30 Sep 11 18:41:52.214861 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:52.214831 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e8a1657-b905-4add-84b2-5549f720a1d2-model-cache\") pod \"5e8a1657-b905-4add-84b2-5549f720a1d2\" (UID: \"5e8a1657-b905-4add-84b2-5549f720a1d2\") " Sep 11 18:41:52.215157 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:52.214878 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e8a1657-b905-4add-84b2-5549f720a1d2-dshm\") pod \"5e8a1657-b905-4add-84b2-5549f720a1d2\" (UID: \"5e8a1657-b905-4add-84b2-5549f720a1d2\") " Sep 11 18:41:52.215157 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:52.214987 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e8a1657-b905-4add-84b2-5549f720a1d2-home\") pod \"5e8a1657-b905-4add-84b2-5549f720a1d2\" (UID: \"5e8a1657-b905-4add-84b2-5549f720a1d2\") " Sep 11 18:41:52.215157 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:52.215077 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e8a1657-b905-4add-84b2-5549f720a1d2-model-cache" (OuterVolumeSpecName: "model-cache") pod "5e8a1657-b905-4add-84b2-5549f720a1d2" (UID: "5e8a1657-b905-4add-84b2-5549f720a1d2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:41:52.215157 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:52.215104 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pmjtx\" (UniqueName: \"kubernetes.io/projected/5e8a1657-b905-4add-84b2-5549f720a1d2-kube-api-access-pmjtx\") pod \"5e8a1657-b905-4add-84b2-5549f720a1d2\" (UID: \"5e8a1657-b905-4add-84b2-5549f720a1d2\") " Sep 11 18:41:52.215157 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:52.215156 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e8a1657-b905-4add-84b2-5549f720a1d2-tmp-dir\") pod \"5e8a1657-b905-4add-84b2-5549f720a1d2\" (UID: \"5e8a1657-b905-4add-84b2-5549f720a1d2\") " Sep 11 18:41:52.215439 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:52.215197 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e8a1657-b905-4add-84b2-5549f720a1d2-tls-certs\") pod \"5e8a1657-b905-4add-84b2-5549f720a1d2\" (UID: \"5e8a1657-b905-4add-84b2-5549f720a1d2\") " Sep 11 18:41:52.215439 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:52.215266 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e8a1657-b905-4add-84b2-5549f720a1d2-home" (OuterVolumeSpecName: "home") pod "5e8a1657-b905-4add-84b2-5549f720a1d2" (UID: "5e8a1657-b905-4add-84b2-5549f720a1d2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:41:52.215439 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:52.215373 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e8a1657-b905-4add-84b2-5549f720a1d2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5e8a1657-b905-4add-84b2-5549f720a1d2" (UID: "5e8a1657-b905-4add-84b2-5549f720a1d2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:41:52.215560 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:52.215534 2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e8a1657-b905-4add-84b2-5549f720a1d2-model-cache\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:41:52.215560 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:52.215551 2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e8a1657-b905-4add-84b2-5549f720a1d2-home\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:41:52.215560 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:52.215559 2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e8a1657-b905-4add-84b2-5549f720a1d2-tmp-dir\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:41:52.217223 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:52.217193 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e8a1657-b905-4add-84b2-5549f720a1d2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5e8a1657-b905-4add-84b2-5549f720a1d2" (UID: "5e8a1657-b905-4add-84b2-5549f720a1d2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:41:52.217328 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:52.217256 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e8a1657-b905-4add-84b2-5549f720a1d2-dshm" (OuterVolumeSpecName: "dshm") pod "5e8a1657-b905-4add-84b2-5549f720a1d2" (UID: "5e8a1657-b905-4add-84b2-5549f720a1d2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:41:52.217328 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:52.217280 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e8a1657-b905-4add-84b2-5549f720a1d2-kube-api-access-pmjtx" (OuterVolumeSpecName: "kube-api-access-pmjtx") pod "5e8a1657-b905-4add-84b2-5549f720a1d2" (UID: "5e8a1657-b905-4add-84b2-5549f720a1d2"). InnerVolumeSpecName "kube-api-access-pmjtx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:41:52.316531 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:52.316493 2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e8a1657-b905-4add-84b2-5549f720a1d2-tls-certs\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:41:52.316531 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:52.316528 2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e8a1657-b905-4add-84b2-5549f720a1d2-dshm\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:41:52.316531 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:52.316537 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmjtx\" (UniqueName: \"kubernetes.io/projected/5e8a1657-b905-4add-84b2-5549f720a1d2-kube-api-access-pmjtx\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:41:52.624461 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:52.624419 2561 generic.go:356] "Generic (PLEG): container finished" podID="5e8a1657-b905-4add-84b2-5549f720a1d2" containerID="f73ee44416f4ffcbe319e6d08535687099e4d7ecdec454822198132e4a3a042f" exitCode=0 Sep 11 18:41:52.624621 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:52.624485 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-efa8b9/embedding-w0-efa8b9-kserve-6d8f694465-l5drh" event={"ID":"5e8a1657-b905-4add-84b2-5549f720a1d2","Type":"ContainerDied","Data":"f73ee44416f4ffcbe319e6d08535687099e4d7ecdec454822198132e4a3a042f"} Sep 11 18:41:52.624621 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:52.624532 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-efa8b9/embedding-w0-efa8b9-kserve-6d8f694465-l5drh" event={"ID":"5e8a1657-b905-4add-84b2-5549f720a1d2","Type":"ContainerDied","Data":"d35e3cb5ce2eb6dac4e4e5ef03031e8e68309966f2c1cda65a7a40693dd6dcde"} Sep 11 18:41:52.624621 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:52.624553 2561 scope.go:117] "RemoveContainer" containerID="f73ee44416f4ffcbe319e6d08535687099e4d7ecdec454822198132e4a3a042f" Sep 11 18:41:52.624621 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:52.624487 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-efa8b9/embedding-w0-efa8b9-kserve-6d8f694465-l5drh" Sep 11 18:41:52.634878 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:52.634856 2561 scope.go:117] "RemoveContainer" containerID="f73ee44416f4ffcbe319e6d08535687099e4d7ecdec454822198132e4a3a042f" Sep 11 18:41:52.635227 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:41:52.635206 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f73ee44416f4ffcbe319e6d08535687099e4d7ecdec454822198132e4a3a042f\": container with ID starting with f73ee44416f4ffcbe319e6d08535687099e4d7ecdec454822198132e4a3a042f not found: ID does not exist" containerID="f73ee44416f4ffcbe319e6d08535687099e4d7ecdec454822198132e4a3a042f" Sep 11 18:41:52.635310 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:52.635241 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f73ee44416f4ffcbe319e6d08535687099e4d7ecdec454822198132e4a3a042f"} err="failed to get container status \"f73ee44416f4ffcbe319e6d08535687099e4d7ecdec454822198132e4a3a042f\": rpc error: code = NotFound desc = could not find container \"f73ee44416f4ffcbe319e6d08535687099e4d7ecdec454822198132e4a3a042f\": container with ID starting with f73ee44416f4ffcbe319e6d08535687099e4d7ecdec454822198132e4a3a042f not found: ID does not exist" Sep 11 18:41:52.649980 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:52.649948 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-efa8b9/embedding-w0-efa8b9-kserve-6d8f694465-l5drh"] Sep 11 18:41:52.657174 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:52.657147 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-efa8b9/embedding-w0-efa8b9-kserve-6d8f694465-l5drh"] Sep 11 18:41:53.140939 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:53.140901 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e8a1657-b905-4add-84b2-5549f720a1d2" path="/var/lib/kubelet/pods/5e8a1657-b905-4add-84b2-5549f720a1d2/volumes" Sep 11 18:41:54.838852 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:54.838815 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-0ead89-65d6c74f65-2tx94"] Sep 11 18:41:54.839305 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:54.839092 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0ead89-65d6c74f65-2tx94" podUID="e2c31ea5-d698-47c7-b149-ee08f7eb184e" containerName="payload-processing" containerID="cri-o://49d69abb877ce8172e13d1a6141e31196269b34460090e142605a9d433db9161" gracePeriod=30 Sep 11 18:41:55.123145 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:55.123112 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0ead89-65d6c74f65-2tx94" Sep 11 18:41:55.251125 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:55.251047 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2c31ea5-d698-47c7-b149-ee08f7eb184e-plugins-config-volume\") pod \"e2c31ea5-d698-47c7-b149-ee08f7eb184e\" (UID: \"e2c31ea5-d698-47c7-b149-ee08f7eb184e\") " Sep 11 18:41:55.251303 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:55.251205 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lff4j\" (UniqueName: \"kubernetes.io/projected/e2c31ea5-d698-47c7-b149-ee08f7eb184e-kube-api-access-lff4j\") pod \"e2c31ea5-d698-47c7-b149-ee08f7eb184e\" (UID: \"e2c31ea5-d698-47c7-b149-ee08f7eb184e\") " Sep 11 18:41:55.251426 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:55.251401 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2c31ea5-d698-47c7-b149-ee08f7eb184e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e2c31ea5-d698-47c7-b149-ee08f7eb184e" (UID: "e2c31ea5-d698-47c7-b149-ee08f7eb184e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:41:55.253442 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:55.253406 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2c31ea5-d698-47c7-b149-ee08f7eb184e-kube-api-access-lff4j" (OuterVolumeSpecName: "kube-api-access-lff4j") pod "e2c31ea5-d698-47c7-b149-ee08f7eb184e" (UID: "e2c31ea5-d698-47c7-b149-ee08f7eb184e"). InnerVolumeSpecName "kube-api-access-lff4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:41:55.352505 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:55.352419 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lff4j\" (UniqueName: \"kubernetes.io/projected/e2c31ea5-d698-47c7-b149-ee08f7eb184e-kube-api-access-lff4j\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:41:55.352505 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:55.352452 2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2c31ea5-d698-47c7-b149-ee08f7eb184e-plugins-config-volume\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:41:55.645910 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:55.645822 2561 generic.go:356] "Generic (PLEG): container finished" podID="e2c31ea5-d698-47c7-b149-ee08f7eb184e" containerID="49d69abb877ce8172e13d1a6141e31196269b34460090e142605a9d433db9161" exitCode=0 Sep 11 18:41:55.645910 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:55.645864 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0ead89-65d6c74f65-2tx94" Sep 11 18:41:55.645910 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:55.645892 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0ead89-65d6c74f65-2tx94" event={"ID":"e2c31ea5-d698-47c7-b149-ee08f7eb184e","Type":"ContainerDied","Data":"49d69abb877ce8172e13d1a6141e31196269b34460090e142605a9d433db9161"} Sep 11 18:41:55.646243 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:55.645943 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-0ead89-65d6c74f65-2tx94" event={"ID":"e2c31ea5-d698-47c7-b149-ee08f7eb184e","Type":"ContainerDied","Data":"aaeb90ca0c952a2c6b6236e890b4881595107dec4c19325b475551b7a6842432"} Sep 11 18:41:55.646243 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:55.645969 2561 scope.go:117] "RemoveContainer" containerID="49d69abb877ce8172e13d1a6141e31196269b34460090e142605a9d433db9161" Sep 11 18:41:55.656877 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:55.656856 2561 scope.go:117] "RemoveContainer" containerID="49d69abb877ce8172e13d1a6141e31196269b34460090e142605a9d433db9161" Sep 11 18:41:55.657227 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:41:55.657203 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49d69abb877ce8172e13d1a6141e31196269b34460090e142605a9d433db9161\": container with ID starting with 49d69abb877ce8172e13d1a6141e31196269b34460090e142605a9d433db9161 not found: ID does not exist" containerID="49d69abb877ce8172e13d1a6141e31196269b34460090e142605a9d433db9161" Sep 11 18:41:55.657278 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:55.657238 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49d69abb877ce8172e13d1a6141e31196269b34460090e142605a9d433db9161"} err="failed to get container status \"49d69abb877ce8172e13d1a6141e31196269b34460090e142605a9d433db9161\": rpc error: code = NotFound desc = could not find container \"49d69abb877ce8172e13d1a6141e31196269b34460090e142605a9d433db9161\": container with ID starting with 49d69abb877ce8172e13d1a6141e31196269b34460090e142605a9d433db9161 not found: ID does not exist" Sep 11 18:41:55.670039 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:55.670008 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-0ead89-65d6c74f65-2tx94"] Sep 11 18:41:55.674566 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:55.674537 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-0ead89-65d6c74f65-2tx94"] Sep 11 18:41:57.140598 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:57.140565 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2c31ea5-d698-47c7-b149-ee08f7eb184e" path="/var/lib/kubelet/pods/e2c31ea5-d698-47c7-b149-ee08f7eb184e/volumes" Sep 11 18:41:57.422155 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:57.422133 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm" Sep 11 18:41:57.575005 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:57.574968 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/891f8576-360b-4004-b612-2169d1323649-workload-socket\") pod \"891f8576-360b-4004-b612-2169d1323649\" (UID: \"891f8576-360b-4004-b612-2169d1323649\") " Sep 11 18:41:57.575215 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:57.575025 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/891f8576-360b-4004-b612-2169d1323649-istio-podinfo\") pod \"891f8576-360b-4004-b612-2169d1323649\" (UID: \"891f8576-360b-4004-b612-2169d1323649\") " Sep 11 18:41:57.575215 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:57.575170 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/891f8576-360b-4004-b612-2169d1323649-istio-envoy\") pod \"891f8576-360b-4004-b612-2169d1323649\" (UID: \"891f8576-360b-4004-b612-2169d1323649\") " Sep 11 18:41:57.575215 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:57.575205 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/891f8576-360b-4004-b612-2169d1323649-workload-certs\") pod \"891f8576-360b-4004-b612-2169d1323649\" (UID: \"891f8576-360b-4004-b612-2169d1323649\") " Sep 11 18:41:57.575323 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:57.575241 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/891f8576-360b-4004-b612-2169d1323649-istiod-ca-cert\") pod \"891f8576-360b-4004-b612-2169d1323649\" (UID: \"891f8576-360b-4004-b612-2169d1323649\") " Sep 11 18:41:57.575323 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:57.575276 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/891f8576-360b-4004-b612-2169d1323649-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "891f8576-360b-4004-b612-2169d1323649" (UID: "891f8576-360b-4004-b612-2169d1323649"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:41:57.575323 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:57.575298 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vdvjd\" (UniqueName: \"kubernetes.io/projected/891f8576-360b-4004-b612-2169d1323649-kube-api-access-vdvjd\") pod \"891f8576-360b-4004-b612-2169d1323649\" (UID: \"891f8576-360b-4004-b612-2169d1323649\") " Sep 11 18:41:57.575470 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:57.575338 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/891f8576-360b-4004-b612-2169d1323649-istio-token\") pod \"891f8576-360b-4004-b612-2169d1323649\" (UID: \"891f8576-360b-4004-b612-2169d1323649\") " Sep 11 18:41:57.575470 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:57.575411 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/891f8576-360b-4004-b612-2169d1323649-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "891f8576-360b-4004-b612-2169d1323649" (UID: "891f8576-360b-4004-b612-2169d1323649"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:41:57.575470 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:57.575437 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/891f8576-360b-4004-b612-2169d1323649-istio-data\") pod \"891f8576-360b-4004-b612-2169d1323649\" (UID: \"891f8576-360b-4004-b612-2169d1323649\") " Sep 11 18:41:57.575637 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:57.575488 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/891f8576-360b-4004-b612-2169d1323649-credential-socket\") pod \"891f8576-360b-4004-b612-2169d1323649\" (UID: \"891f8576-360b-4004-b612-2169d1323649\") " Sep 11 18:41:57.575697 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:57.575631 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/891f8576-360b-4004-b612-2169d1323649-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "891f8576-360b-4004-b612-2169d1323649" (UID: "891f8576-360b-4004-b612-2169d1323649"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:41:57.575697 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:57.575685 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/891f8576-360b-4004-b612-2169d1323649-istio-data" (OuterVolumeSpecName: "istio-data") pod "891f8576-360b-4004-b612-2169d1323649" (UID: "891f8576-360b-4004-b612-2169d1323649"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:41:57.575819 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:57.575776 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/891f8576-360b-4004-b612-2169d1323649-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "891f8576-360b-4004-b612-2169d1323649" (UID: "891f8576-360b-4004-b612-2169d1323649"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:41:57.575925 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:57.575909 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/891f8576-360b-4004-b612-2169d1323649-workload-certs\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:41:57.575988 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:57.575926 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/891f8576-360b-4004-b612-2169d1323649-istiod-ca-cert\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:41:57.575988 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:57.575938 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/891f8576-360b-4004-b612-2169d1323649-istio-data\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:41:57.575988 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:57.575950 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/891f8576-360b-4004-b612-2169d1323649-credential-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:41:57.575988 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:57.575964 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/891f8576-360b-4004-b612-2169d1323649-workload-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:41:57.577344 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:57.577315 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/891f8576-360b-4004-b612-2169d1323649-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "891f8576-360b-4004-b612-2169d1323649" (UID: "891f8576-360b-4004-b612-2169d1323649"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:41:57.577482 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:57.577465 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/891f8576-360b-4004-b612-2169d1323649-istio-token" (OuterVolumeSpecName: "istio-token") pod "891f8576-360b-4004-b612-2169d1323649" (UID: "891f8576-360b-4004-b612-2169d1323649"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:41:57.577704 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:57.577683 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/891f8576-360b-4004-b612-2169d1323649-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "891f8576-360b-4004-b612-2169d1323649" (UID: "891f8576-360b-4004-b612-2169d1323649"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 18:41:57.577763 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:57.577719 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/891f8576-360b-4004-b612-2169d1323649-kube-api-access-vdvjd" (OuterVolumeSpecName: "kube-api-access-vdvjd") pod "891f8576-360b-4004-b612-2169d1323649" (UID: "891f8576-360b-4004-b612-2169d1323649"). InnerVolumeSpecName "kube-api-access-vdvjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:41:57.661801 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:57.661766 2561 generic.go:356] "Generic (PLEG): container finished" podID="891f8576-360b-4004-b612-2169d1323649" containerID="9db63a3867e2f67cc94e3282540a39d880f3cdca10d549c085bd50cceb730fdc" exitCode=0 Sep 11 18:41:57.662003 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:57.661816 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm" Sep 11 18:41:57.662003 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:57.661830 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm" event={"ID":"891f8576-360b-4004-b612-2169d1323649","Type":"ContainerDied","Data":"9db63a3867e2f67cc94e3282540a39d880f3cdca10d549c085bd50cceb730fdc"} Sep 11 18:41:57.662003 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:57.661919 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm" event={"ID":"891f8576-360b-4004-b612-2169d1323649","Type":"ContainerDied","Data":"a2e3a32b355e18cb481e56708a64c8050cfc8da199c6ff0b4919d3ccbd0033b6"} Sep 11 18:41:57.662003 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:57.661966 2561 scope.go:117] "RemoveContainer" containerID="9db63a3867e2f67cc94e3282540a39d880f3cdca10d549c085bd50cceb730fdc" Sep 11 18:41:57.672210 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:57.672191 2561 scope.go:117] "RemoveContainer" containerID="7f53dc95d5b8ed078e3362f1f885c8b4f0ba38761ee23614e8b82934138015f7" Sep 11 18:41:57.676822 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:57.676796 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdvjd\" (UniqueName: \"kubernetes.io/projected/891f8576-360b-4004-b612-2169d1323649-kube-api-access-vdvjd\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:41:57.676822 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:57.676821 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/891f8576-360b-4004-b612-2169d1323649-istio-token\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:41:57.676953 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:57.676831 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/891f8576-360b-4004-b612-2169d1323649-istio-podinfo\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:41:57.676953 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:57.676839 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/891f8576-360b-4004-b612-2169d1323649-istio-envoy\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:41:57.681885 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:57.681864 2561 scope.go:117] "RemoveContainer" containerID="9db63a3867e2f67cc94e3282540a39d880f3cdca10d549c085bd50cceb730fdc" Sep 11 18:41:57.682246 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:41:57.682222 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9db63a3867e2f67cc94e3282540a39d880f3cdca10d549c085bd50cceb730fdc\": container with ID starting with 9db63a3867e2f67cc94e3282540a39d880f3cdca10d549c085bd50cceb730fdc not found: ID does not exist" containerID="9db63a3867e2f67cc94e3282540a39d880f3cdca10d549c085bd50cceb730fdc" Sep 11 18:41:57.682331 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:57.682259 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9db63a3867e2f67cc94e3282540a39d880f3cdca10d549c085bd50cceb730fdc"} err="failed to get container status \"9db63a3867e2f67cc94e3282540a39d880f3cdca10d549c085bd50cceb730fdc\": rpc error: code = NotFound desc = could not find container \"9db63a3867e2f67cc94e3282540a39d880f3cdca10d549c085bd50cceb730fdc\": container with ID starting with 9db63a3867e2f67cc94e3282540a39d880f3cdca10d549c085bd50cceb730fdc not found: ID does not exist" Sep 11 18:41:57.682331 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:57.682284 2561 scope.go:117] "RemoveContainer" containerID="7f53dc95d5b8ed078e3362f1f885c8b4f0ba38761ee23614e8b82934138015f7" Sep 11 18:41:57.682586 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:41:57.682566 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f53dc95d5b8ed078e3362f1f885c8b4f0ba38761ee23614e8b82934138015f7\": container with ID starting with 7f53dc95d5b8ed078e3362f1f885c8b4f0ba38761ee23614e8b82934138015f7 not found: ID does not exist" containerID="7f53dc95d5b8ed078e3362f1f885c8b4f0ba38761ee23614e8b82934138015f7" Sep 11 18:41:57.682646 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:57.682593 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f53dc95d5b8ed078e3362f1f885c8b4f0ba38761ee23614e8b82934138015f7"} err="failed to get container status \"7f53dc95d5b8ed078e3362f1f885c8b4f0ba38761ee23614e8b82934138015f7\": rpc error: code = NotFound desc = could not find container \"7f53dc95d5b8ed078e3362f1f885c8b4f0ba38761ee23614e8b82934138015f7\": container with ID starting with 7f53dc95d5b8ed078e3362f1f885c8b4f0ba38761ee23614e8b82934138015f7 not found: ID does not exist" Sep 11 18:41:57.693760 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:57.693729 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm"] Sep 11 18:41:57.698774 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:57.698743 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-efa8b9-openshift-default-66754d5974-fqvsm"] Sep 11 18:41:59.139975 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:41:59.139888 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="891f8576-360b-4004-b612-2169d1323649" path="/var/lib/kubelet/pods/891f8576-360b-4004-b612-2169d1323649/volumes" Sep 11 18:42:02.005302 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:02.005266 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc"] Sep 11 18:42:02.005735 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:02.005540 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc" podUID="b79f3a4c-779f-4157-9746-bfe9348292b3" containerName="istio-proxy" containerID="cri-o://7be3b41565610a3e0e08e86292685ae57342ef764f0690126b1d15cf1f216ad2" gracePeriod=30 Sep 11 18:42:07.262620 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:07.262592 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc" Sep 11 18:42:07.369504 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:07.369411 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b79f3a4c-779f-4157-9746-bfe9348292b3-credential-socket\") pod \"b79f3a4c-779f-4157-9746-bfe9348292b3\" (UID: \"b79f3a4c-779f-4157-9746-bfe9348292b3\") " Sep 11 18:42:07.369504 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:07.369485 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b79f3a4c-779f-4157-9746-bfe9348292b3-istiod-ca-cert\") pod \"b79f3a4c-779f-4157-9746-bfe9348292b3\" (UID: \"b79f3a4c-779f-4157-9746-bfe9348292b3\") " Sep 11 18:42:07.369741 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:07.369525 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b79f3a4c-779f-4157-9746-bfe9348292b3-istio-data\") pod \"b79f3a4c-779f-4157-9746-bfe9348292b3\" (UID: \"b79f3a4c-779f-4157-9746-bfe9348292b3\") " Sep 11 18:42:07.369741 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:07.369555 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b79f3a4c-779f-4157-9746-bfe9348292b3-workload-certs\") pod \"b79f3a4c-779f-4157-9746-bfe9348292b3\" (UID: \"b79f3a4c-779f-4157-9746-bfe9348292b3\") " Sep 11 18:42:07.369741 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:07.369617 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d4sjs\" (UniqueName: \"kubernetes.io/projected/b79f3a4c-779f-4157-9746-bfe9348292b3-kube-api-access-d4sjs\") pod \"b79f3a4c-779f-4157-9746-bfe9348292b3\" (UID: \"b79f3a4c-779f-4157-9746-bfe9348292b3\") " Sep 11 18:42:07.369741 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:07.369658 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b79f3a4c-779f-4157-9746-bfe9348292b3-workload-socket\") pod \"b79f3a4c-779f-4157-9746-bfe9348292b3\" (UID: \"b79f3a4c-779f-4157-9746-bfe9348292b3\") " Sep 11 18:42:07.369741 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:07.369680 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b79f3a4c-779f-4157-9746-bfe9348292b3-istio-token\") pod \"b79f3a4c-779f-4157-9746-bfe9348292b3\" (UID: \"b79f3a4c-779f-4157-9746-bfe9348292b3\") " Sep 11 18:42:07.369741 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:07.369706 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b79f3a4c-779f-4157-9746-bfe9348292b3-istio-envoy\") pod \"b79f3a4c-779f-4157-9746-bfe9348292b3\" (UID: \"b79f3a4c-779f-4157-9746-bfe9348292b3\") " Sep 11 18:42:07.370027 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:07.369710 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b79f3a4c-779f-4157-9746-bfe9348292b3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b79f3a4c-779f-4157-9746-bfe9348292b3" (UID: "b79f3a4c-779f-4157-9746-bfe9348292b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:42:07.370027 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:07.369751 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b79f3a4c-779f-4157-9746-bfe9348292b3-istio-podinfo\") pod \"b79f3a4c-779f-4157-9746-bfe9348292b3\" (UID: \"b79f3a4c-779f-4157-9746-bfe9348292b3\") " Sep 11 18:42:07.370027 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:07.369791 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b79f3a4c-779f-4157-9746-bfe9348292b3-istio-data" (OuterVolumeSpecName: "istio-data") pod "b79f3a4c-779f-4157-9746-bfe9348292b3" (UID: "b79f3a4c-779f-4157-9746-bfe9348292b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:42:07.370027 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:07.369860 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b79f3a4c-779f-4157-9746-bfe9348292b3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b79f3a4c-779f-4157-9746-bfe9348292b3" (UID: "b79f3a4c-779f-4157-9746-bfe9348292b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:42:07.370027 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:07.369918 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b79f3a4c-779f-4157-9746-bfe9348292b3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b79f3a4c-779f-4157-9746-bfe9348292b3" (UID: "b79f3a4c-779f-4157-9746-bfe9348292b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:42:07.370264 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:07.370170 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b79f3a4c-779f-4157-9746-bfe9348292b3-credential-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:42:07.370324 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:07.370306 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b79f3a4c-779f-4157-9746-bfe9348292b3-istiod-ca-cert\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:42:07.370380 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:07.370324 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b79f3a4c-779f-4157-9746-bfe9348292b3-istio-data\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:42:07.370380 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:07.370337 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b79f3a4c-779f-4157-9746-bfe9348292b3-workload-certs\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:42:07.370380 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:07.370235 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b79f3a4c-779f-4157-9746-bfe9348292b3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b79f3a4c-779f-4157-9746-bfe9348292b3" (UID: "b79f3a4c-779f-4157-9746-bfe9348292b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:42:07.371842 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:07.371816 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b79f3a4c-779f-4157-9746-bfe9348292b3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b79f3a4c-779f-4157-9746-bfe9348292b3" (UID: "b79f3a4c-779f-4157-9746-bfe9348292b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 18:42:07.371971 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:07.371908 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b79f3a4c-779f-4157-9746-bfe9348292b3-kube-api-access-d4sjs" (OuterVolumeSpecName: "kube-api-access-d4sjs") pod "b79f3a4c-779f-4157-9746-bfe9348292b3" (UID: "b79f3a4c-779f-4157-9746-bfe9348292b3"). InnerVolumeSpecName "kube-api-access-d4sjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:42:07.372017 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:07.371977 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b79f3a4c-779f-4157-9746-bfe9348292b3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b79f3a4c-779f-4157-9746-bfe9348292b3" (UID: "b79f3a4c-779f-4157-9746-bfe9348292b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:42:07.372071 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:07.372020 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b79f3a4c-779f-4157-9746-bfe9348292b3-istio-token" (OuterVolumeSpecName: "istio-token") pod "b79f3a4c-779f-4157-9746-bfe9348292b3" (UID: "b79f3a4c-779f-4157-9746-bfe9348292b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:42:07.471521 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:07.471482 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4sjs\" (UniqueName: \"kubernetes.io/projected/b79f3a4c-779f-4157-9746-bfe9348292b3-kube-api-access-d4sjs\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:42:07.471521 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:07.471515 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b79f3a4c-779f-4157-9746-bfe9348292b3-workload-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:42:07.471521 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:07.471525 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b79f3a4c-779f-4157-9746-bfe9348292b3-istio-token\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:42:07.471762 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:07.471533 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b79f3a4c-779f-4157-9746-bfe9348292b3-istio-envoy\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:42:07.471762 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:07.471541 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b79f3a4c-779f-4157-9746-bfe9348292b3-istio-podinfo\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:42:07.728193 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:07.728153 2561 generic.go:356] "Generic (PLEG): container finished" podID="b79f3a4c-779f-4157-9746-bfe9348292b3" containerID="7be3b41565610a3e0e08e86292685ae57342ef764f0690126b1d15cf1f216ad2" exitCode=0 Sep 11 18:42:07.728401 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:07.728215 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc" Sep 11 18:42:07.728401 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:07.728250 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc" event={"ID":"b79f3a4c-779f-4157-9746-bfe9348292b3","Type":"ContainerDied","Data":"7be3b41565610a3e0e08e86292685ae57342ef764f0690126b1d15cf1f216ad2"} Sep 11 18:42:07.728401 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:07.728320 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc" event={"ID":"b79f3a4c-779f-4157-9746-bfe9348292b3","Type":"ContainerDied","Data":"5dbbff5d7bc809dd9780b20521863e58cdc46a7f98810c0435e2c633f53068a0"} Sep 11 18:42:07.728401 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:07.728353 2561 scope.go:117] "RemoveContainer" containerID="7be3b41565610a3e0e08e86292685ae57342ef764f0690126b1d15cf1f216ad2" Sep 11 18:42:07.738949 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:07.738930 2561 scope.go:117] "RemoveContainer" containerID="64677ebfe10d115e7faddd914b169ff5d6921b41ba0b7b4c31cc8ee39fe1f91b" Sep 11 18:42:07.750780 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:07.750757 2561 scope.go:117] "RemoveContainer" containerID="7be3b41565610a3e0e08e86292685ae57342ef764f0690126b1d15cf1f216ad2" Sep 11 18:42:07.751375 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:42:07.751327 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7be3b41565610a3e0e08e86292685ae57342ef764f0690126b1d15cf1f216ad2\": container with ID starting with 7be3b41565610a3e0e08e86292685ae57342ef764f0690126b1d15cf1f216ad2 not found: ID does not exist" containerID="7be3b41565610a3e0e08e86292685ae57342ef764f0690126b1d15cf1f216ad2" Sep 11 18:42:07.751482 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:07.751380 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7be3b41565610a3e0e08e86292685ae57342ef764f0690126b1d15cf1f216ad2"} err="failed to get container status \"7be3b41565610a3e0e08e86292685ae57342ef764f0690126b1d15cf1f216ad2\": rpc error: code = NotFound desc = could not find container \"7be3b41565610a3e0e08e86292685ae57342ef764f0690126b1d15cf1f216ad2\": container with ID starting with 7be3b41565610a3e0e08e86292685ae57342ef764f0690126b1d15cf1f216ad2 not found: ID does not exist" Sep 11 18:42:07.751482 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:07.751398 2561 scope.go:117] "RemoveContainer" containerID="64677ebfe10d115e7faddd914b169ff5d6921b41ba0b7b4c31cc8ee39fe1f91b" Sep 11 18:42:07.751915 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:42:07.751878 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64677ebfe10d115e7faddd914b169ff5d6921b41ba0b7b4c31cc8ee39fe1f91b\": container with ID starting with 64677ebfe10d115e7faddd914b169ff5d6921b41ba0b7b4c31cc8ee39fe1f91b not found: ID does not exist" containerID="64677ebfe10d115e7faddd914b169ff5d6921b41ba0b7b4c31cc8ee39fe1f91b" Sep 11 18:42:07.752106 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:07.751920 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64677ebfe10d115e7faddd914b169ff5d6921b41ba0b7b4c31cc8ee39fe1f91b"} err="failed to get container status \"64677ebfe10d115e7faddd914b169ff5d6921b41ba0b7b4c31cc8ee39fe1f91b\": rpc error: code = NotFound desc = could not find container \"64677ebfe10d115e7faddd914b169ff5d6921b41ba0b7b4c31cc8ee39fe1f91b\": container with ID starting with 64677ebfe10d115e7faddd914b169ff5d6921b41ba0b7b4c31cc8ee39fe1f91b not found: ID does not exist" Sep 11 18:42:07.755595 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:07.755571 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc"] Sep 11 18:42:07.761928 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:07.761889 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-0ead89-openshift-default-dd4fb94fd-w9fsc"] Sep 11 18:42:09.140008 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:09.139970 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b79f3a4c-779f-4157-9746-bfe9348292b3" path="/var/lib/kubelet/pods/b79f3a4c-779f-4157-9746-bfe9348292b3/volumes" Sep 11 18:42:19.764604 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:19.764483 2561 scope.go:117] "RemoveContainer" containerID="6de739ac64002a6ef60650d5bfedd1f00cc9efef65a305dae1c3c14d14d6ec0d" Sep 11 18:42:21.560376 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:21.560335 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7638-056d25"] Sep 11 18:42:21.560806 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:21.560791 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2df4550-54e5-42d1-9257-077917d8d1b3" containerName="istio-proxy" Sep 11 18:42:21.560846 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:21.560808 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2df4550-54e5-42d1-9257-077917d8d1b3" containerName="istio-proxy" Sep 11 18:42:21.560846 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:21.560820 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="891f8576-360b-4004-b612-2169d1323649" containerName="istio-proxy" Sep 11 18:42:21.560846 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:21.560825 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="891f8576-360b-4004-b612-2169d1323649" containerName="istio-proxy" Sep 11 18:42:21.560846 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:21.560831 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="489ae584-3686-403e-bd3b-797e3f78868d" containerName="maas-api" Sep 11 18:42:21.560846 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:21.560837 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="489ae584-3686-403e-bd3b-797e3f78868d" containerName="maas-api" Sep 11 18:42:21.560846 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:21.560846 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ecbed32-91f0-44b6-aa50-2ed6f61bdc98" containerName="istio-proxy" Sep 11 18:42:21.561050 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:21.560853 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ecbed32-91f0-44b6-aa50-2ed6f61bdc98" containerName="istio-proxy" Sep 11 18:42:21.561050 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:21.560861 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2c31ea5-d698-47c7-b149-ee08f7eb184e" containerName="payload-processing" Sep 11 18:42:21.561050 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:21.560866 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2c31ea5-d698-47c7-b149-ee08f7eb184e" containerName="payload-processing" Sep 11 18:42:21.561050 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:21.560878 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e8a1657-b905-4add-84b2-5549f720a1d2" containerName="main" Sep 11 18:42:21.561050 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:21.560884 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e8a1657-b905-4add-84b2-5549f720a1d2" containerName="main" Sep 11 18:42:21.561050 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:21.560889 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b79f3a4c-779f-4157-9746-bfe9348292b3" containerName="istio-proxy" Sep 11 18:42:21.561050 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:21.560894 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b79f3a4c-779f-4157-9746-bfe9348292b3" containerName="istio-proxy" Sep 11 18:42:21.561050 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:21.560905 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b4839f5-19a5-4f78-88d5-99d06b629f30" containerName="payload-pre-processing" Sep 11 18:42:21.561050 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:21.560910 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b4839f5-19a5-4f78-88d5-99d06b629f30" containerName="payload-pre-processing" Sep 11 18:42:21.561050 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:21.560917 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ecbed32-91f0-44b6-aa50-2ed6f61bdc98" containerName="istio-proxy" Sep 11 18:42:21.561050 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:21.560922 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ecbed32-91f0-44b6-aa50-2ed6f61bdc98" containerName="istio-proxy" Sep 11 18:42:21.561050 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:21.560931 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b79f3a4c-779f-4157-9746-bfe9348292b3" containerName="istio-proxy" Sep 11 18:42:21.561050 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:21.560936 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b79f3a4c-779f-4157-9746-bfe9348292b3" containerName="istio-proxy" Sep 11 18:42:21.561050 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:21.560946 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ae31332-0851-4f28-b6ad-096f486ce60a" containerName="payload-processing" Sep 11 18:42:21.561050 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:21.560951 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ae31332-0851-4f28-b6ad-096f486ce60a" containerName="payload-processing" Sep 11 18:42:21.561050 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:21.561013 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2df4550-54e5-42d1-9257-077917d8d1b3" containerName="istio-proxy" Sep 11 18:42:21.561050 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:21.561019 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b4839f5-19a5-4f78-88d5-99d06b629f30" containerName="payload-pre-processing" Sep 11 18:42:21.561050 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:21.561028 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="489ae584-3686-403e-bd3b-797e3f78868d" containerName="maas-api" Sep 11 18:42:21.561050 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:21.561034 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b79f3a4c-779f-4157-9746-bfe9348292b3" containerName="istio-proxy" Sep 11 18:42:21.561050 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:21.561041 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ecbed32-91f0-44b6-aa50-2ed6f61bdc98" containerName="istio-proxy" Sep 11 18:42:21.561050 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:21.561046 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b79f3a4c-779f-4157-9746-bfe9348292b3" containerName="istio-proxy" Sep 11 18:42:21.561050 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:21.561079 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2df4550-54e5-42d1-9257-077917d8d1b3" containerName="istio-proxy" Sep 11 18:42:21.561050 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:21.561086 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ae31332-0851-4f28-b6ad-096f486ce60a" containerName="payload-processing" Sep 11 18:42:21.561763 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:21.561094 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="891f8576-360b-4004-b612-2169d1323649" containerName="istio-proxy" Sep 11 18:42:21.561763 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:21.561099 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="891f8576-360b-4004-b612-2169d1323649" containerName="istio-proxy" Sep 11 18:42:21.561763 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:21.561104 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ecbed32-91f0-44b6-aa50-2ed6f61bdc98" containerName="istio-proxy" Sep 11 18:42:21.561763 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:21.561109 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e8a1657-b905-4add-84b2-5549f720a1d2" containerName="main" Sep 11 18:42:21.561763 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:21.561115 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2c31ea5-d698-47c7-b149-ee08f7eb184e" containerName="payload-processing" Sep 11 18:42:21.563858 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:21.563842 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7638-056d25" Sep 11 18:42:21.566002 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:21.565980 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-bbqnn\"" Sep 11 18:42:21.572742 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:21.572715 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7638-056d25"] Sep 11 18:42:21.721765 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:21.721731 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w275s\" (UniqueName: \"kubernetes.io/projected/9f798015-6020-47e2-a7d0-bdb35e96af47-kube-api-access-w275s\") pod \"test-curl-7638-056d25\" (UID: \"9f798015-6020-47e2-a7d0-bdb35e96af47\") " pod="opendatahub/test-curl-7638-056d25" Sep 11 18:42:21.822421 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:21.822323 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w275s\" (UniqueName: \"kubernetes.io/projected/9f798015-6020-47e2-a7d0-bdb35e96af47-kube-api-access-w275s\") pod \"test-curl-7638-056d25\" (UID: \"9f798015-6020-47e2-a7d0-bdb35e96af47\") " pod="opendatahub/test-curl-7638-056d25" Sep 11 18:42:21.832251 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:21.832217 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w275s\" (UniqueName: \"kubernetes.io/projected/9f798015-6020-47e2-a7d0-bdb35e96af47-kube-api-access-w275s\") pod \"test-curl-7638-056d25\" (UID: \"9f798015-6020-47e2-a7d0-bdb35e96af47\") " pod="opendatahub/test-curl-7638-056d25" Sep 11 18:42:21.875314 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:21.875277 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7638-056d25" Sep 11 18:42:22.155822 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:22.155790 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7638-056d25"] Sep 11 18:42:22.156964 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:42:22.156937 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f798015_6020_47e2_a7d0_bdb35e96af47.slice/crio-e2e1bfac10818a6ffb0500a9ed19da78b237e8e0e601367a21c63761ede6844e WatchSource:0}: Error finding container e2e1bfac10818a6ffb0500a9ed19da78b237e8e0e601367a21c63761ede6844e: Status 404 returned error can't find the container with id e2e1bfac10818a6ffb0500a9ed19da78b237e8e0e601367a21c63761ede6844e Sep 11 18:42:22.831870 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:22.831828 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7638-056d25" event={"ID":"9f798015-6020-47e2-a7d0-bdb35e96af47","Type":"ContainerStarted","Data":"2ba4463ed4a49d22346e3de8bfccdf9f115983975a0c0d34c12624ab28cf47cc"} Sep 11 18:42:22.832291 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:22.831884 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7638-056d25" event={"ID":"9f798015-6020-47e2-a7d0-bdb35e96af47","Type":"ContainerStarted","Data":"e2e1bfac10818a6ffb0500a9ed19da78b237e8e0e601367a21c63761ede6844e"} Sep 11 18:42:22.850602 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:22.850542 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7638-056d25" podStartSLOduration=1.850525513 podStartE2EDuration="1.850525513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:42:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:42:22.847373465 +0000 UTC m=+1624.344391218" watchObservedRunningTime="2026-09-11 18:42:22.850525513 +0000 UTC m=+1624.347543266" Sep 11 18:42:23.431872 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:23.431834 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7638-056d25"] Sep 11 18:42:23.439721 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:23.439689 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7638-056d25"] Sep 11 18:42:23.838034 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:42:23.838001 2561 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7638-056d25" container="test-curl-7638-056d25" Sep 11 18:42:24.844329 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:24.844291 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7638-056d25" podUID="9f798015-6020-47e2-a7d0-bdb35e96af47" containerName="test-curl-7638-056d25" containerID="cri-o://2ba4463ed4a49d22346e3de8bfccdf9f115983975a0c0d34c12624ab28cf47cc" gracePeriod=2 Sep 11 18:42:24.846249 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:42:24.846215 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7638-056d25\" is forbidden: User \"system:node:ip-10-0-138-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-128.ec2.internal' and this object" podUID="9f798015-6020-47e2-a7d0-bdb35e96af47" pod="opendatahub/test-curl-7638-056d25" Sep 11 18:42:27.039601 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:27.039567 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7638-056d25" Sep 11 18:42:27.175810 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:27.175768 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w275s\" (UniqueName: \"kubernetes.io/projected/9f798015-6020-47e2-a7d0-bdb35e96af47-kube-api-access-w275s\") pod \"9f798015-6020-47e2-a7d0-bdb35e96af47\" (UID: \"9f798015-6020-47e2-a7d0-bdb35e96af47\") " Sep 11 18:42:27.177848 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:27.177813 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f798015-6020-47e2-a7d0-bdb35e96af47-kube-api-access-w275s" (OuterVolumeSpecName: "kube-api-access-w275s") pod "9f798015-6020-47e2-a7d0-bdb35e96af47" (UID: "9f798015-6020-47e2-a7d0-bdb35e96af47"). InnerVolumeSpecName "kube-api-access-w275s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:42:27.277068 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:27.277026 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w275s\" (UniqueName: \"kubernetes.io/projected/9f798015-6020-47e2-a7d0-bdb35e96af47-kube-api-access-w275s\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:42:27.865438 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:27.865395 2561 generic.go:356] "Generic (PLEG): container finished" podID="9f798015-6020-47e2-a7d0-bdb35e96af47" containerID="2ba4463ed4a49d22346e3de8bfccdf9f115983975a0c0d34c12624ab28cf47cc" exitCode=137 Sep 11 18:42:27.865438 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:27.865436 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7638-056d25" Sep 11 18:42:27.865691 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:27.865478 2561 scope.go:117] "RemoveContainer" containerID="2ba4463ed4a49d22346e3de8bfccdf9f115983975a0c0d34c12624ab28cf47cc" Sep 11 18:42:27.874815 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:27.874796 2561 scope.go:117] "RemoveContainer" containerID="2ba4463ed4a49d22346e3de8bfccdf9f115983975a0c0d34c12624ab28cf47cc" Sep 11 18:42:27.875096 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:42:27.875075 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ba4463ed4a49d22346e3de8bfccdf9f115983975a0c0d34c12624ab28cf47cc\": container with ID starting with 2ba4463ed4a49d22346e3de8bfccdf9f115983975a0c0d34c12624ab28cf47cc not found: ID does not exist" containerID="2ba4463ed4a49d22346e3de8bfccdf9f115983975a0c0d34c12624ab28cf47cc" Sep 11 18:42:27.875150 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:27.875106 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ba4463ed4a49d22346e3de8bfccdf9f115983975a0c0d34c12624ab28cf47cc"} err="failed to get container status \"2ba4463ed4a49d22346e3de8bfccdf9f115983975a0c0d34c12624ab28cf47cc\": rpc error: code = NotFound desc = could not find container \"2ba4463ed4a49d22346e3de8bfccdf9f115983975a0c0d34c12624ab28cf47cc\": container with ID starting with 2ba4463ed4a49d22346e3de8bfccdf9f115983975a0c0d34c12624ab28cf47cc not found: ID does not exist" Sep 11 18:42:29.139662 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:29.139625 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f798015-6020-47e2-a7d0-bdb35e96af47" path="/var/lib/kubelet/pods/9f798015-6020-47e2-a7d0-bdb35e96af47/volumes" Sep 11 18:42:32.642078 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:32.642006 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-724327-29819190-dq289"] Sep 11 18:42:32.650042 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:32.650009 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-724327-29819190-dq289"] Sep 11 18:42:33.139296 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:42:33.139263 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc23f483-c0dc-48d1-9a0c-5376be955b73" path="/var/lib/kubelet/pods/fc23f483-c0dc-48d1-9a0c-5376be955b73/volumes" Sep 11 18:43:19.818115 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:43:19.818080 2561 scope.go:117] "RemoveContainer" containerID="22ad7edaabd914d42156437e73507ccd1b24c6dc79d988a7f3f9b76889778d6f" Sep 11 18:44:16.050765 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:44:16.050728 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7638-a0bd0e"] Sep 11 18:44:16.051391 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:44:16.051367 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f798015-6020-47e2-a7d0-bdb35e96af47" containerName="test-curl-7638-056d25" Sep 11 18:44:16.051479 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:44:16.051392 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f798015-6020-47e2-a7d0-bdb35e96af47" containerName="test-curl-7638-056d25" Sep 11 18:44:16.051479 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:44:16.051412 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2df4550-54e5-42d1-9257-077917d8d1b3" containerName="istio-proxy" Sep 11 18:44:16.051479 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:44:16.051420 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2df4550-54e5-42d1-9257-077917d8d1b3" containerName="istio-proxy" Sep 11 18:44:16.051479 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:44:16.051437 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="891f8576-360b-4004-b612-2169d1323649" containerName="istio-proxy" Sep 11 18:44:16.051479 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:44:16.051447 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="891f8576-360b-4004-b612-2169d1323649" containerName="istio-proxy" Sep 11 18:44:16.051730 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:44:16.051548 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f798015-6020-47e2-a7d0-bdb35e96af47" containerName="test-curl-7638-056d25" Sep 11 18:44:16.054599 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:44:16.054577 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7638-a0bd0e" Sep 11 18:44:16.057156 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:44:16.057129 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-bbqnn\"" Sep 11 18:44:16.061196 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:44:16.061170 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7638-a0bd0e"] Sep 11 18:44:16.120509 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:44:16.120472 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9l6x\" (UniqueName: \"kubernetes.io/projected/75a0b5c0-8bce-4ac1-bde6-d264e021a632-kube-api-access-w9l6x\") pod \"test-curl-7638-a0bd0e\" (UID: \"75a0b5c0-8bce-4ac1-bde6-d264e021a632\") " pod="opendatahub/test-curl-7638-a0bd0e" Sep 11 18:44:16.221302 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:44:16.221264 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9l6x\" (UniqueName: \"kubernetes.io/projected/75a0b5c0-8bce-4ac1-bde6-d264e021a632-kube-api-access-w9l6x\") pod \"test-curl-7638-a0bd0e\" (UID: \"75a0b5c0-8bce-4ac1-bde6-d264e021a632\") " pod="opendatahub/test-curl-7638-a0bd0e" Sep 11 18:44:16.230264 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:44:16.230234 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9l6x\" (UniqueName: \"kubernetes.io/projected/75a0b5c0-8bce-4ac1-bde6-d264e021a632-kube-api-access-w9l6x\") pod \"test-curl-7638-a0bd0e\" (UID: \"75a0b5c0-8bce-4ac1-bde6-d264e021a632\") " pod="opendatahub/test-curl-7638-a0bd0e" Sep 11 18:44:16.365609 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:44:16.365523 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7638-a0bd0e" Sep 11 18:44:16.638418 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:44:16.638383 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7638-a0bd0e"] Sep 11 18:44:16.639802 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:44:16.639775 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75a0b5c0_8bce_4ac1_bde6_d264e021a632.slice/crio-d70c52178ec11e6493fe5f508cbde4c11a0cf24979320366be45c8097717c89e WatchSource:0}: Error finding container d70c52178ec11e6493fe5f508cbde4c11a0cf24979320366be45c8097717c89e: Status 404 returned error can't find the container with id d70c52178ec11e6493fe5f508cbde4c11a0cf24979320366be45c8097717c89e Sep 11 18:44:17.555003 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:44:17.554961 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7638-a0bd0e" event={"ID":"75a0b5c0-8bce-4ac1-bde6-d264e021a632","Type":"ContainerStarted","Data":"606840bf6523669f8e603a311c7a52674c94c94145b219082635517f64aca409"} Sep 11 18:44:17.555003 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:44:17.555007 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7638-a0bd0e" event={"ID":"75a0b5c0-8bce-4ac1-bde6-d264e021a632","Type":"ContainerStarted","Data":"d70c52178ec11e6493fe5f508cbde4c11a0cf24979320366be45c8097717c89e"} Sep 11 18:44:17.570886 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:44:17.570829 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7638-a0bd0e" podStartSLOduration=1.570809695 podStartE2EDuration="1.570809695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:44:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:44:17.570110655 +0000 UTC m=+1739.067128409" watchObservedRunningTime="2026-09-11 18:44:17.570809695 +0000 UTC m=+1739.067827465" Sep 11 18:44:18.134618 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:44:18.134581 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7638-a0bd0e"] Sep 11 18:44:18.139432 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:44:18.139401 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7638-a0bd0e"] Sep 11 18:44:18.561880 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:44:18.561847 2561 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7638-a0bd0e" container="test-curl-7638-a0bd0e" Sep 11 18:44:19.568103 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:44:19.568050 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7638-a0bd0e" podUID="75a0b5c0-8bce-4ac1-bde6-d264e021a632" containerName="test-curl-7638-a0bd0e" containerID="cri-o://606840bf6523669f8e603a311c7a52674c94c94145b219082635517f64aca409" gracePeriod=2 Sep 11 18:44:21.717163 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:44:21.717138 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7638-a0bd0e" Sep 11 18:44:21.773996 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:44:21.773964 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9l6x\" (UniqueName: \"kubernetes.io/projected/75a0b5c0-8bce-4ac1-bde6-d264e021a632-kube-api-access-w9l6x\") pod \"75a0b5c0-8bce-4ac1-bde6-d264e021a632\" (UID: \"75a0b5c0-8bce-4ac1-bde6-d264e021a632\") " Sep 11 18:44:21.775994 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:44:21.775967 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75a0b5c0-8bce-4ac1-bde6-d264e021a632-kube-api-access-w9l6x" (OuterVolumeSpecName: "kube-api-access-w9l6x") pod "75a0b5c0-8bce-4ac1-bde6-d264e021a632" (UID: "75a0b5c0-8bce-4ac1-bde6-d264e021a632"). InnerVolumeSpecName "kube-api-access-w9l6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:44:21.874972 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:44:21.874891 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9l6x\" (UniqueName: \"kubernetes.io/projected/75a0b5c0-8bce-4ac1-bde6-d264e021a632-kube-api-access-w9l6x\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:44:22.588844 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:44:22.588812 2561 generic.go:356] "Generic (PLEG): container finished" podID="75a0b5c0-8bce-4ac1-bde6-d264e021a632" containerID="606840bf6523669f8e603a311c7a52674c94c94145b219082635517f64aca409" exitCode=137 Sep 11 18:44:22.589028 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:44:22.588866 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7638-a0bd0e" Sep 11 18:44:22.589028 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:44:22.588891 2561 scope.go:117] "RemoveContainer" containerID="606840bf6523669f8e603a311c7a52674c94c94145b219082635517f64aca409" Sep 11 18:44:22.601022 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:44:22.600999 2561 scope.go:117] "RemoveContainer" containerID="606840bf6523669f8e603a311c7a52674c94c94145b219082635517f64aca409" Sep 11 18:44:22.601418 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:44:22.601395 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"606840bf6523669f8e603a311c7a52674c94c94145b219082635517f64aca409\": container with ID starting with 606840bf6523669f8e603a311c7a52674c94c94145b219082635517f64aca409 not found: ID does not exist" containerID="606840bf6523669f8e603a311c7a52674c94c94145b219082635517f64aca409" Sep 11 18:44:22.601469 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:44:22.601423 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7638-a0bd0e\" is forbidden: User \"system:node:ip-10-0-138-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-128.ec2.internal' and this object" podUID="75a0b5c0-8bce-4ac1-bde6-d264e021a632" pod="opendatahub/test-curl-7638-a0bd0e" Sep 11 18:44:22.601469 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:44:22.601429 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"606840bf6523669f8e603a311c7a52674c94c94145b219082635517f64aca409"} err="failed to get container status \"606840bf6523669f8e603a311c7a52674c94c94145b219082635517f64aca409\": rpc error: code = NotFound desc = could not find container \"606840bf6523669f8e603a311c7a52674c94c94145b219082635517f64aca409\": container with ID starting with 606840bf6523669f8e603a311c7a52674c94c94145b219082635517f64aca409 not found: ID does not exist" Sep 11 18:44:23.139815 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:44:23.139785 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75a0b5c0-8bce-4ac1-bde6-d264e021a632" path="/var/lib/kubelet/pods/75a0b5c0-8bce-4ac1-bde6-d264e021a632/volumes" Sep 11 18:45:19.534796 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:45:19.534760 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dj524_60702ef3-98ef-463b-b877-3a00825353d8/console-operator/1.log" Sep 11 18:45:19.542987 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:45:19.542964 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4knd_42e5ddd8-a96c-45a1-adf9-caa47ee902db/ovn-acl-logging/0.log" Sep 11 18:45:19.544835 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:45:19.544814 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dj524_60702ef3-98ef-463b-b877-3a00825353d8/console-operator/1.log" Sep 11 18:45:19.552681 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:45:19.552660 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4knd_42e5ddd8-a96c-45a1-adf9-caa47ee902db/ovn-acl-logging/0.log" Sep 11 18:46:13.604484 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:46:13.604433 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-946072-7445dfc668-pm9lv"] Sep 11 18:46:13.605100 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:46:13.605082 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75a0b5c0-8bce-4ac1-bde6-d264e021a632" containerName="test-curl-7638-a0bd0e" Sep 11 18:46:13.605152 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:46:13.605103 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="75a0b5c0-8bce-4ac1-bde6-d264e021a632" containerName="test-curl-7638-a0bd0e" Sep 11 18:46:13.605236 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:46:13.605225 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="75a0b5c0-8bce-4ac1-bde6-d264e021a632" containerName="test-curl-7638-a0bd0e" Sep 11 18:46:13.608904 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:46:13.608879 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-946072-7445dfc668-pm9lv" Sep 11 18:46:13.611115 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:46:13.611081 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-946072-serving-cert\"" Sep 11 18:46:13.617457 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:46:13.617409 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-946072-7445dfc668-pm9lv"] Sep 11 18:46:13.674512 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:46:13.674468 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e80d9a66-2f1b-4f88-b1c1-3cd31958d700-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-946072-7445dfc668-pm9lv\" (UID: \"e80d9a66-2f1b-4f88-b1c1-3cd31958d700\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-946072-7445dfc668-pm9lv" Sep 11 18:46:13.674512 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:46:13.674510 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e80d9a66-2f1b-4f88-b1c1-3cd31958d700-metrics-certs\") pod \"maas-api-e2e-ipp-b-946072-7445dfc668-pm9lv\" (UID: \"e80d9a66-2f1b-4f88-b1c1-3cd31958d700\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-946072-7445dfc668-pm9lv" Sep 11 18:46:13.674727 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:46:13.674535 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e80d9a66-2f1b-4f88-b1c1-3cd31958d700-maas-api-tls\") pod \"maas-api-e2e-ipp-b-946072-7445dfc668-pm9lv\" (UID: \"e80d9a66-2f1b-4f88-b1c1-3cd31958d700\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-946072-7445dfc668-pm9lv" Sep 11 18:46:13.674727 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:46:13.674556 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vwql\" (UniqueName: \"kubernetes.io/projected/e80d9a66-2f1b-4f88-b1c1-3cd31958d700-kube-api-access-9vwql\") pod \"maas-api-e2e-ipp-b-946072-7445dfc668-pm9lv\" (UID: \"e80d9a66-2f1b-4f88-b1c1-3cd31958d700\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-946072-7445dfc668-pm9lv" Sep 11 18:46:13.775734 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:46:13.775695 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e80d9a66-2f1b-4f88-b1c1-3cd31958d700-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-946072-7445dfc668-pm9lv\" (UID: \"e80d9a66-2f1b-4f88-b1c1-3cd31958d700\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-946072-7445dfc668-pm9lv" Sep 11 18:46:13.775734 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:46:13.775735 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e80d9a66-2f1b-4f88-b1c1-3cd31958d700-metrics-certs\") pod \"maas-api-e2e-ipp-b-946072-7445dfc668-pm9lv\" (UID: \"e80d9a66-2f1b-4f88-b1c1-3cd31958d700\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-946072-7445dfc668-pm9lv" Sep 11 18:46:13.775985 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:46:13.775780 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e80d9a66-2f1b-4f88-b1c1-3cd31958d700-maas-api-tls\") pod \"maas-api-e2e-ipp-b-946072-7445dfc668-pm9lv\" (UID: \"e80d9a66-2f1b-4f88-b1c1-3cd31958d700\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-946072-7445dfc668-pm9lv" Sep 11 18:46:13.775985 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:46:13.775827 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vwql\" (UniqueName: \"kubernetes.io/projected/e80d9a66-2f1b-4f88-b1c1-3cd31958d700-kube-api-access-9vwql\") pod \"maas-api-e2e-ipp-b-946072-7445dfc668-pm9lv\" (UID: \"e80d9a66-2f1b-4f88-b1c1-3cd31958d700\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-946072-7445dfc668-pm9lv" Sep 11 18:46:13.776461 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:46:13.776429 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e80d9a66-2f1b-4f88-b1c1-3cd31958d700-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-946072-7445dfc668-pm9lv\" (UID: \"e80d9a66-2f1b-4f88-b1c1-3cd31958d700\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-946072-7445dfc668-pm9lv" Sep 11 18:46:13.778473 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:46:13.778445 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e80d9a66-2f1b-4f88-b1c1-3cd31958d700-metrics-certs\") pod \"maas-api-e2e-ipp-b-946072-7445dfc668-pm9lv\" (UID: \"e80d9a66-2f1b-4f88-b1c1-3cd31958d700\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-946072-7445dfc668-pm9lv" Sep 11 18:46:13.778601 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:46:13.778545 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e80d9a66-2f1b-4f88-b1c1-3cd31958d700-maas-api-tls\") pod \"maas-api-e2e-ipp-b-946072-7445dfc668-pm9lv\" (UID: \"e80d9a66-2f1b-4f88-b1c1-3cd31958d700\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-946072-7445dfc668-pm9lv" Sep 11 18:46:13.784799 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:46:13.784767 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vwql\" (UniqueName: \"kubernetes.io/projected/e80d9a66-2f1b-4f88-b1c1-3cd31958d700-kube-api-access-9vwql\") pod \"maas-api-e2e-ipp-b-946072-7445dfc668-pm9lv\" (UID: \"e80d9a66-2f1b-4f88-b1c1-3cd31958d700\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-946072-7445dfc668-pm9lv" Sep 11 18:46:13.925803 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:46:13.925761 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-946072-7445dfc668-pm9lv" Sep 11 18:46:14.213803 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:46:14.213763 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-946072-7445dfc668-pm9lv"] Sep 11 18:46:14.214738 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:46:14.214709 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode80d9a66_2f1b_4f88_b1c1_3cd31958d700.slice/crio-e05aabbde81723651ec316341c991855ffb3b0075e05b847f8853bdb1af1bebc WatchSource:0}: Error finding container e05aabbde81723651ec316341c991855ffb3b0075e05b847f8853bdb1af1bebc: Status 404 returned error can't find the container with id e05aabbde81723651ec316341c991855ffb3b0075e05b847f8853bdb1af1bebc Sep 11 18:46:14.216020 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:46:14.216000 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 18:46:14.295773 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:46:14.295728 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-946072-7445dfc668-pm9lv" event={"ID":"e80d9a66-2f1b-4f88-b1c1-3cd31958d700","Type":"ContainerStarted","Data":"e05aabbde81723651ec316341c991855ffb3b0075e05b847f8853bdb1af1bebc"} Sep 11 18:46:15.304145 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:46:15.304104 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-946072-7445dfc668-pm9lv" event={"ID":"e80d9a66-2f1b-4f88-b1c1-3cd31958d700","Type":"ContainerStarted","Data":"dfff36c336dd1fcee8fddf1ae442309115d774f75010a613e7ac9f08a816fd1f"} Sep 11 18:46:15.304576 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:46:15.304234 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-946072-7445dfc668-pm9lv" Sep 11 18:46:15.321505 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:46:15.321438 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-946072-7445dfc668-pm9lv" podStartSLOduration=1.910283559 podStartE2EDuration="2.321418972s" totalImagesPullingTime="411.135413ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:46:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:46:15.319789182 +0000 UTC m=+1856.816806936" watchObservedRunningTime="2026-09-11 18:46:15.321418972 +0000 UTC m=+1856.818436727" Sep 11 18:46:21.316490 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:46:21.316460 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-946072-7445dfc668-pm9lv" Sep 11 18:47:12.618260 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:12.618224 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9"] Sep 11 18:47:12.622167 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:12.622142 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9" Sep 11 18:47:12.624419 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:12.624395 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-e9c09a-openshift-default-dockercfg-mtkmk\"" Sep 11 18:47:12.633573 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:12.633536 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9"] Sep 11 18:47:12.762332 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:12.762292 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f-istio-envoy\") pod \"e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9\" (UID: \"2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f\") " pod="openshift-ingress/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9" Sep 11 18:47:12.762520 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:12.762357 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f-istio-data\") pod \"e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9\" (UID: \"2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f\") " pod="openshift-ingress/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9" Sep 11 18:47:12.762520 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:12.762388 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f-workload-socket\") pod \"e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9\" (UID: \"2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f\") " pod="openshift-ingress/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9" Sep 11 18:47:12.762520 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:12.762425 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f-istio-token\") pod \"e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9\" (UID: \"2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f\") " pod="openshift-ingress/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9" Sep 11 18:47:12.762520 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:12.762487 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f-istio-podinfo\") pod \"e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9\" (UID: \"2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f\") " pod="openshift-ingress/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9" Sep 11 18:47:12.762694 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:12.762557 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f-credential-socket\") pod \"e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9\" (UID: \"2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f\") " pod="openshift-ingress/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9" Sep 11 18:47:12.762694 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:12.762572 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f-workload-certs\") pod \"e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9\" (UID: \"2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f\") " pod="openshift-ingress/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9" Sep 11 18:47:12.762694 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:12.762600 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4c6v\" (UniqueName: \"kubernetes.io/projected/2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f-kube-api-access-w4c6v\") pod \"e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9\" (UID: \"2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f\") " pod="openshift-ingress/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9" Sep 11 18:47:12.762694 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:12.762647 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f-istiod-ca-cert\") pod \"e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9\" (UID: \"2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f\") " pod="openshift-ingress/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9" Sep 11 18:47:12.863429 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:12.863393 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f-istiod-ca-cert\") pod \"e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9\" (UID: \"2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f\") " pod="openshift-ingress/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9" Sep 11 18:47:12.863646 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:12.863478 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f-istio-envoy\") pod \"e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9\" (UID: \"2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f\") " pod="openshift-ingress/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9" Sep 11 18:47:12.863646 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:12.863523 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f-istio-data\") pod \"e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9\" (UID: \"2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f\") " pod="openshift-ingress/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9" Sep 11 18:47:12.863646 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:12.863547 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f-workload-socket\") pod \"e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9\" (UID: \"2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f\") " pod="openshift-ingress/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9" Sep 11 18:47:12.863646 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:12.863582 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f-istio-token\") pod \"e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9\" (UID: \"2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f\") " pod="openshift-ingress/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9" Sep 11 18:47:12.863646 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:12.863624 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f-istio-podinfo\") pod \"e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9\" (UID: \"2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f\") " pod="openshift-ingress/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9" Sep 11 18:47:12.863925 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:12.863670 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f-credential-socket\") pod \"e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9\" (UID: \"2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f\") " pod="openshift-ingress/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9" Sep 11 18:47:12.863925 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:12.863696 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f-workload-certs\") pod \"e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9\" (UID: \"2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f\") " pod="openshift-ingress/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9" Sep 11 18:47:12.863925 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:12.863725 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4c6v\" (UniqueName: \"kubernetes.io/projected/2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f-kube-api-access-w4c6v\") pod \"e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9\" (UID: \"2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f\") " pod="openshift-ingress/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9" Sep 11 18:47:12.864124 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:12.863947 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f-istio-data\") pod \"e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9\" (UID: \"2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f\") " pod="openshift-ingress/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9" Sep 11 18:47:12.864124 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:12.863998 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f-workload-socket\") pod \"e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9\" (UID: \"2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f\") " pod="openshift-ingress/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9" Sep 11 18:47:12.864241 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:12.864179 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f-credential-socket\") pod \"e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9\" (UID: \"2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f\") " pod="openshift-ingress/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9" Sep 11 18:47:12.864296 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:12.864234 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f-workload-certs\") pod \"e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9\" (UID: \"2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f\") " pod="openshift-ingress/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9" Sep 11 18:47:12.864296 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:12.864270 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f-istiod-ca-cert\") pod \"e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9\" (UID: \"2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f\") " pod="openshift-ingress/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9" Sep 11 18:47:12.865940 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:12.865915 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f-istio-envoy\") pod \"e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9\" (UID: \"2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f\") " pod="openshift-ingress/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9" Sep 11 18:47:12.866416 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:12.866397 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f-istio-podinfo\") pod \"e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9\" (UID: \"2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f\") " pod="openshift-ingress/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9" Sep 11 18:47:12.873594 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:12.873525 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f-istio-token\") pod \"e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9\" (UID: \"2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f\") " pod="openshift-ingress/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9" Sep 11 18:47:12.873594 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:12.873565 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4c6v\" (UniqueName: \"kubernetes.io/projected/2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f-kube-api-access-w4c6v\") pod \"e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9\" (UID: \"2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f\") " pod="openshift-ingress/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9" Sep 11 18:47:12.935330 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:12.935299 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9" Sep 11 18:47:13.221823 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:13.221756 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9"] Sep 11 18:47:13.222892 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:47:13.222865 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2756ee4d_8f93_4182_9c1e_2a4b56fa8a1f.slice/crio-ffdeb4d4fb437cab6f10e0f33c47e1b9ddc94c9d09a1b81f4ab9aedb8e4a77f9 WatchSource:0}: Error finding container ffdeb4d4fb437cab6f10e0f33c47e1b9ddc94c9d09a1b81f4ab9aedb8e4a77f9: Status 404 returned error can't find the container with id ffdeb4d4fb437cab6f10e0f33c47e1b9ddc94c9d09a1b81f4ab9aedb8e4a77f9 Sep 11 18:47:13.225715 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:13.225670 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:47:13.225824 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:13.225757 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:47:13.225824 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:13.225808 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:47:13.660401 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:13.660365 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9" event={"ID":"2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f","Type":"ContainerStarted","Data":"2e5f009423a1a7cc71d25efcb335dad9fcca3f09a87e532fdedb1b7aa703d25b"} Sep 11 18:47:13.660811 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:13.660426 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9" event={"ID":"2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f","Type":"ContainerStarted","Data":"ffdeb4d4fb437cab6f10e0f33c47e1b9ddc94c9d09a1b81f4ab9aedb8e4a77f9"} Sep 11 18:47:13.683216 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:13.683157 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9" podStartSLOduration=1.68313977 podStartE2EDuration="1.68313977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:47:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:47:13.680072159 +0000 UTC m=+1915.177089903" watchObservedRunningTime="2026-09-11 18:47:13.68313977 +0000 UTC m=+1915.180157569" Sep 11 18:47:13.935444 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:13.935368 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9" Sep 11 18:47:14.941652 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:14.941618 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9" Sep 11 18:47:15.674700 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:15.674666 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9" Sep 11 18:47:15.675528 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:15.675507 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9" Sep 11 18:47:35.805145 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:35.805106 2561 generic.go:356] "Generic (PLEG): container finished" podID="2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f" containerID="2e5f009423a1a7cc71d25efcb335dad9fcca3f09a87e532fdedb1b7aa703d25b" exitCode=0 Sep 11 18:47:35.805570 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:35.805183 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9" event={"ID":"2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f","Type":"ContainerDied","Data":"2e5f009423a1a7cc71d25efcb335dad9fcca3f09a87e532fdedb1b7aa703d25b"} Sep 11 18:47:35.805616 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:35.805584 2561 scope.go:117] "RemoveContainer" containerID="2e5f009423a1a7cc71d25efcb335dad9fcca3f09a87e532fdedb1b7aa703d25b" Sep 11 18:47:35.806707 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:35.806674 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:47:35.806810 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:35.806747 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:47:35.806810 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:35.806782 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 11 18:47:36.814238 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:36.814167 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9" event={"ID":"2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f","Type":"ContainerStarted","Data":"d67ff08db6b289236954f7ca366c2cfb7674286eac1c00af4e87107e78e536ba"} Sep 11 18:47:36.935884 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:36.935849 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9" Sep 11 18:47:36.936111 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:36.936088 2561 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Sep 11 18:47:36.936166 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:36.936138 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9" podUID="2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Sep 11 18:47:37.935775 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:37.935737 2561 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Sep 11 18:47:37.936166 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:37.935790 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9" podUID="2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Sep 11 18:47:38.935967 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:38.935924 2561 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Sep 11 18:47:38.936387 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:38.936023 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9" podUID="2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Sep 11 18:47:39.935846 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:39.935808 2561 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Sep 11 18:47:39.936023 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:39.935869 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9" podUID="2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Sep 11 18:47:40.936028 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:40.935988 2561 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Sep 11 18:47:40.936443 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:40.936048 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9" podUID="2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Sep 11 18:47:41.936627 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:41.936585 2561 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Sep 11 18:47:41.937019 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:41.936650 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9" podUID="2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Sep 11 18:47:43.001565 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:43.001523 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9" Sep 11 18:47:43.002216 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:43.002184 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9" Sep 11 18:47:43.003778 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:47:43.003754 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9" Sep 11 18:48:04.849311 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:48:04.849274 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9"] Sep 11 18:48:04.849828 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:48:04.849511 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9" podUID="2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f" containerName="istio-proxy" containerID="cri-o://d67ff08db6b289236954f7ca366c2cfb7674286eac1c00af4e87107e78e536ba" gracePeriod=30 Sep 11 18:48:10.034196 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:48:10.034162 2561 generic.go:356] "Generic (PLEG): container finished" podID="2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f" containerID="d67ff08db6b289236954f7ca366c2cfb7674286eac1c00af4e87107e78e536ba" exitCode=0 Sep 11 18:48:10.034675 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:48:10.034213 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9" event={"ID":"2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f","Type":"ContainerDied","Data":"d67ff08db6b289236954f7ca366c2cfb7674286eac1c00af4e87107e78e536ba"} Sep 11 18:48:10.034675 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:48:10.034261 2561 scope.go:117] "RemoveContainer" containerID="2e5f009423a1a7cc71d25efcb335dad9fcca3f09a87e532fdedb1b7aa703d25b" Sep 11 18:48:10.127300 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:48:10.127221 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9" Sep 11 18:48:10.258711 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:48:10.258670 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f-istio-podinfo\") pod \"2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f\" (UID: \"2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f\") " Sep 11 18:48:10.258711 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:48:10.258720 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w4c6v\" (UniqueName: \"kubernetes.io/projected/2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f-kube-api-access-w4c6v\") pod \"2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f\" (UID: \"2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f\") " Sep 11 18:48:10.258975 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:48:10.258760 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f-credential-socket\") pod \"2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f\" (UID: \"2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f\") " Sep 11 18:48:10.258975 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:48:10.258780 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f-istiod-ca-cert\") pod \"2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f\" (UID: \"2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f\") " Sep 11 18:48:10.258975 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:48:10.258916 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f-workload-socket\") pod \"2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f\" (UID: \"2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f\") " Sep 11 18:48:10.259210 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:48:10.258979 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f-workload-certs\") pod \"2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f\" (UID: \"2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f\") " Sep 11 18:48:10.259210 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:48:10.259009 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f-istio-envoy\") pod \"2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f\" (UID: \"2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f\") " Sep 11 18:48:10.259210 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:48:10.259093 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f-istio-token\") pod \"2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f\" (UID: \"2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f\") " Sep 11 18:48:10.259210 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:48:10.259117 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f" (UID: "2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:48:10.259210 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:48:10.259178 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f-istio-data\") pod \"2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f\" (UID: \"2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f\") " Sep 11 18:48:10.259489 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:48:10.259219 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f" (UID: "2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:48:10.259489 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:48:10.259211 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f" (UID: "2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:48:10.259489 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:48:10.259276 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f" (UID: "2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:48:10.259707 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:48:10.259503 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f-istio-data" (OuterVolumeSpecName: "istio-data") pod "2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f" (UID: "2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:48:10.259707 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:48:10.259699 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f-workload-certs\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:48:10.259821 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:48:10.259722 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f-istio-data\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:48:10.259821 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:48:10.259736 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f-credential-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:48:10.259821 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:48:10.259753 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f-istiod-ca-cert\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:48:10.259821 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:48:10.259767 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f-workload-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:48:10.261231 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:48:10.261196 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f" (UID: "2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 18:48:10.261351 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:48:10.261248 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f" (UID: "2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:48:10.261351 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:48:10.261278 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f-kube-api-access-w4c6v" (OuterVolumeSpecName: "kube-api-access-w4c6v") pod "2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f" (UID: "2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f"). InnerVolumeSpecName "kube-api-access-w4c6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:48:10.261429 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:48:10.261398 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f-istio-token" (OuterVolumeSpecName: "istio-token") pod "2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f" (UID: "2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:48:10.360452 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:48:10.360410 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f-istio-token\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:48:10.360452 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:48:10.360448 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f-istio-podinfo\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:48:10.360452 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:48:10.360458 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4c6v\" (UniqueName: \"kubernetes.io/projected/2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f-kube-api-access-w4c6v\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:48:10.360703 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:48:10.360477 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f-istio-envoy\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:48:11.042350 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:48:11.042319 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9" Sep 11 18:48:11.042804 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:48:11.042360 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9" event={"ID":"2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f","Type":"ContainerDied","Data":"ffdeb4d4fb437cab6f10e0f33c47e1b9ddc94c9d09a1b81f4ab9aedb8e4a77f9"} Sep 11 18:48:11.042804 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:48:11.042426 2561 scope.go:117] "RemoveContainer" containerID="d67ff08db6b289236954f7ca366c2cfb7674286eac1c00af4e87107e78e536ba" Sep 11 18:48:11.067425 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:48:11.067386 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9"] Sep 11 18:48:11.076009 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:48:11.075978 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e9c09a-openshift-default-f9f8db76b-45wn9"] Sep 11 18:48:11.139389 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:48:11.139358 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f" path="/var/lib/kubelet/pods/2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f/volumes" Sep 11 18:49:29.839240 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:29.839200 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-946072-7445dfc668-pm9lv"] Sep 11 18:49:29.839862 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:29.839469 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-946072-7445dfc668-pm9lv" podUID="e80d9a66-2f1b-4f88-b1c1-3cd31958d700" containerName="maas-api" containerID="cri-o://dfff36c336dd1fcee8fddf1ae442309115d774f75010a613e7ac9f08a816fd1f" gracePeriod=30 Sep 11 18:49:30.099364 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:30.099292 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-946072-7445dfc668-pm9lv" Sep 11 18:49:30.232525 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:30.232487 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9vwql\" (UniqueName: \"kubernetes.io/projected/e80d9a66-2f1b-4f88-b1c1-3cd31958d700-kube-api-access-9vwql\") pod \"e80d9a66-2f1b-4f88-b1c1-3cd31958d700\" (UID: \"e80d9a66-2f1b-4f88-b1c1-3cd31958d700\") " Sep 11 18:49:30.232739 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:30.232604 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e80d9a66-2f1b-4f88-b1c1-3cd31958d700-maas-api-tls\") pod \"e80d9a66-2f1b-4f88-b1c1-3cd31958d700\" (UID: \"e80d9a66-2f1b-4f88-b1c1-3cd31958d700\") " Sep 11 18:49:30.232739 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:30.232651 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e80d9a66-2f1b-4f88-b1c1-3cd31958d700-metrics-certs\") pod \"e80d9a66-2f1b-4f88-b1c1-3cd31958d700\" (UID: \"e80d9a66-2f1b-4f88-b1c1-3cd31958d700\") " Sep 11 18:49:30.232739 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:30.232720 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e80d9a66-2f1b-4f88-b1c1-3cd31958d700-openshift-service-ca\") pod \"e80d9a66-2f1b-4f88-b1c1-3cd31958d700\" (UID: \"e80d9a66-2f1b-4f88-b1c1-3cd31958d700\") " Sep 11 18:49:30.233199 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:30.233113 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e80d9a66-2f1b-4f88-b1c1-3cd31958d700-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e80d9a66-2f1b-4f88-b1c1-3cd31958d700" (UID: "e80d9a66-2f1b-4f88-b1c1-3cd31958d700"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:49:30.233739 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:30.233385 2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e80d9a66-2f1b-4f88-b1c1-3cd31958d700-openshift-service-ca\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:49:30.235325 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:30.235274 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e80d9a66-2f1b-4f88-b1c1-3cd31958d700-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e80d9a66-2f1b-4f88-b1c1-3cd31958d700" (UID: "e80d9a66-2f1b-4f88-b1c1-3cd31958d700"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:49:30.235325 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:30.235280 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e80d9a66-2f1b-4f88-b1c1-3cd31958d700-kube-api-access-9vwql" (OuterVolumeSpecName: "kube-api-access-9vwql") pod "e80d9a66-2f1b-4f88-b1c1-3cd31958d700" (UID: "e80d9a66-2f1b-4f88-b1c1-3cd31958d700"). InnerVolumeSpecName "kube-api-access-9vwql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:49:30.235502 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:30.235425 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e80d9a66-2f1b-4f88-b1c1-3cd31958d700-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e80d9a66-2f1b-4f88-b1c1-3cd31958d700" (UID: "e80d9a66-2f1b-4f88-b1c1-3cd31958d700"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:49:30.334371 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:30.334332 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vwql\" (UniqueName: \"kubernetes.io/projected/e80d9a66-2f1b-4f88-b1c1-3cd31958d700-kube-api-access-9vwql\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:49:30.334371 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:30.334365 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e80d9a66-2f1b-4f88-b1c1-3cd31958d700-maas-api-tls\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:49:30.334371 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:30.334375 2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e80d9a66-2f1b-4f88-b1c1-3cd31958d700-metrics-certs\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:49:30.537340 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:30.537309 2561 generic.go:356] "Generic (PLEG): container finished" podID="e80d9a66-2f1b-4f88-b1c1-3cd31958d700" containerID="dfff36c336dd1fcee8fddf1ae442309115d774f75010a613e7ac9f08a816fd1f" exitCode=0 Sep 11 18:49:30.537550 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:30.537370 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-946072-7445dfc668-pm9lv" Sep 11 18:49:30.537550 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:30.537381 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-946072-7445dfc668-pm9lv" event={"ID":"e80d9a66-2f1b-4f88-b1c1-3cd31958d700","Type":"ContainerDied","Data":"dfff36c336dd1fcee8fddf1ae442309115d774f75010a613e7ac9f08a816fd1f"} Sep 11 18:49:30.537550 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:30.537423 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-946072-7445dfc668-pm9lv" event={"ID":"e80d9a66-2f1b-4f88-b1c1-3cd31958d700","Type":"ContainerDied","Data":"e05aabbde81723651ec316341c991855ffb3b0075e05b847f8853bdb1af1bebc"} Sep 11 18:49:30.537550 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:30.537445 2561 scope.go:117] "RemoveContainer" containerID="dfff36c336dd1fcee8fddf1ae442309115d774f75010a613e7ac9f08a816fd1f" Sep 11 18:49:30.549346 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:30.549256 2561 scope.go:117] "RemoveContainer" containerID="dfff36c336dd1fcee8fddf1ae442309115d774f75010a613e7ac9f08a816fd1f" Sep 11 18:49:30.551845 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:49:30.551814 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfff36c336dd1fcee8fddf1ae442309115d774f75010a613e7ac9f08a816fd1f\": container with ID starting with dfff36c336dd1fcee8fddf1ae442309115d774f75010a613e7ac9f08a816fd1f not found: ID does not exist" containerID="dfff36c336dd1fcee8fddf1ae442309115d774f75010a613e7ac9f08a816fd1f" Sep 11 18:49:30.551946 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:30.551856 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfff36c336dd1fcee8fddf1ae442309115d774f75010a613e7ac9f08a816fd1f"} err="failed to get container status \"dfff36c336dd1fcee8fddf1ae442309115d774f75010a613e7ac9f08a816fd1f\": rpc error: code = NotFound desc = could not find container \"dfff36c336dd1fcee8fddf1ae442309115d774f75010a613e7ac9f08a816fd1f\": container with ID starting with dfff36c336dd1fcee8fddf1ae442309115d774f75010a613e7ac9f08a816fd1f not found: ID does not exist" Sep 11 18:49:30.563183 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:30.563157 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-946072-7445dfc668-pm9lv"] Sep 11 18:49:30.567831 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:30.567806 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-946072-7445dfc668-pm9lv"] Sep 11 18:49:31.139732 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:31.139698 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e80d9a66-2f1b-4f88-b1c1-3cd31958d700" path="/var/lib/kubelet/pods/e80d9a66-2f1b-4f88-b1c1-3cd31958d700/volumes" Sep 11 18:49:47.862535 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:47.862497 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7ca6df-5c896694c5-qf4zs"] Sep 11 18:49:47.863005 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:47.862748 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7ca6df-5c896694c5-qf4zs" podUID="0b4dec75-b53b-4e05-82eb-553ac25bee7d" containerName="maas-api" containerID="cri-o://a0aa8bb0edadd64029c0a08152571f20b5498c55317088b468c82cb13d8f67dc" gracePeriod=30 Sep 11 18:49:47.914323 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:47.914286 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7ca6df-5c896694c5-qf4zs" podUID="0b4dec75-b53b-4e05-82eb-553ac25bee7d" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.96:8443/health\": dial tcp 10.132.0.96:8443: connect: connection refused" Sep 11 18:49:47.934175 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:47.934144 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-7ca6df-6cb6cb5765-c86n8"] Sep 11 18:49:47.934402 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:47.934372 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7ca6df-6cb6cb5765-c86n8" podUID="0ec34a6b-f340-4743-a761-b5327b9e7516" containerName="payload-processing" containerID="cri-o://7047447bfa9db574277d479bc02bd507aaacfada7df24f1f94ab4851bba3d70f" gracePeriod=30 Sep 11 18:49:48.185316 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:48.185289 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7ca6df-5c896694c5-qf4zs" Sep 11 18:49:48.233567 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:48.233544 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7ca6df-6cb6cb5765-c86n8" Sep 11 18:49:48.308987 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:48.308957 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6n9gw\" (UniqueName: \"kubernetes.io/projected/0b4dec75-b53b-4e05-82eb-553ac25bee7d-kube-api-access-6n9gw\") pod \"0b4dec75-b53b-4e05-82eb-553ac25bee7d\" (UID: \"0b4dec75-b53b-4e05-82eb-553ac25bee7d\") " Sep 11 18:49:48.309198 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:48.308999 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9fxh8\" (UniqueName: \"kubernetes.io/projected/0ec34a6b-f340-4743-a761-b5327b9e7516-kube-api-access-9fxh8\") pod \"0ec34a6b-f340-4743-a761-b5327b9e7516\" (UID: \"0ec34a6b-f340-4743-a761-b5327b9e7516\") " Sep 11 18:49:48.309198 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:48.309029 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b4dec75-b53b-4e05-82eb-553ac25bee7d-metrics-certs\") pod \"0b4dec75-b53b-4e05-82eb-553ac25bee7d\" (UID: \"0b4dec75-b53b-4e05-82eb-553ac25bee7d\") " Sep 11 18:49:48.309198 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:48.309093 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b4dec75-b53b-4e05-82eb-553ac25bee7d-openshift-service-ca\") pod \"0b4dec75-b53b-4e05-82eb-553ac25bee7d\" (UID: \"0b4dec75-b53b-4e05-82eb-553ac25bee7d\") " Sep 11 18:49:48.309198 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:48.309140 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ec34a6b-f340-4743-a761-b5327b9e7516-plugins-config-volume\") pod \"0ec34a6b-f340-4743-a761-b5327b9e7516\" (UID: \"0ec34a6b-f340-4743-a761-b5327b9e7516\") " Sep 11 18:49:48.309198 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:48.309173 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b4dec75-b53b-4e05-82eb-553ac25bee7d-maas-api-tls\") pod \"0b4dec75-b53b-4e05-82eb-553ac25bee7d\" (UID: \"0b4dec75-b53b-4e05-82eb-553ac25bee7d\") " Sep 11 18:49:48.309499 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:48.309471 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b4dec75-b53b-4e05-82eb-553ac25bee7d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0b4dec75-b53b-4e05-82eb-553ac25bee7d" (UID: "0b4dec75-b53b-4e05-82eb-553ac25bee7d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:49:48.309581 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:48.309560 2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b4dec75-b53b-4e05-82eb-553ac25bee7d-openshift-service-ca\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:49:48.309646 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:48.309590 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ec34a6b-f340-4743-a761-b5327b9e7516-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0ec34a6b-f340-4743-a761-b5327b9e7516" (UID: "0ec34a6b-f340-4743-a761-b5327b9e7516"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:49:48.311210 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:48.311175 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b4dec75-b53b-4e05-82eb-553ac25bee7d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0b4dec75-b53b-4e05-82eb-553ac25bee7d" (UID: "0b4dec75-b53b-4e05-82eb-553ac25bee7d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:49:48.311346 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:48.311260 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b4dec75-b53b-4e05-82eb-553ac25bee7d-kube-api-access-6n9gw" (OuterVolumeSpecName: "kube-api-access-6n9gw") pod "0b4dec75-b53b-4e05-82eb-553ac25bee7d" (UID: "0b4dec75-b53b-4e05-82eb-553ac25bee7d"). InnerVolumeSpecName "kube-api-access-6n9gw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:49:48.311346 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:48.311326 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ec34a6b-f340-4743-a761-b5327b9e7516-kube-api-access-9fxh8" (OuterVolumeSpecName: "kube-api-access-9fxh8") pod "0ec34a6b-f340-4743-a761-b5327b9e7516" (UID: "0ec34a6b-f340-4743-a761-b5327b9e7516"). InnerVolumeSpecName "kube-api-access-9fxh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:49:48.311434 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:48.311420 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b4dec75-b53b-4e05-82eb-553ac25bee7d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0b4dec75-b53b-4e05-82eb-553ac25bee7d" (UID: "0b4dec75-b53b-4e05-82eb-553ac25bee7d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:49:48.410383 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:48.410357 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n9gw\" (UniqueName: \"kubernetes.io/projected/0b4dec75-b53b-4e05-82eb-553ac25bee7d-kube-api-access-6n9gw\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:49:48.410383 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:48.410382 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fxh8\" (UniqueName: \"kubernetes.io/projected/0ec34a6b-f340-4743-a761-b5327b9e7516-kube-api-access-9fxh8\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:49:48.410556 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:48.410393 2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b4dec75-b53b-4e05-82eb-553ac25bee7d-metrics-certs\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:49:48.410556 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:48.410401 2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ec34a6b-f340-4743-a761-b5327b9e7516-plugins-config-volume\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:49:48.410556 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:48.410410 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b4dec75-b53b-4e05-82eb-553ac25bee7d-maas-api-tls\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:49:48.646913 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:48.646874 2561 generic.go:356] "Generic (PLEG): container finished" podID="0ec34a6b-f340-4743-a761-b5327b9e7516" containerID="7047447bfa9db574277d479bc02bd507aaacfada7df24f1f94ab4851bba3d70f" exitCode=0 Sep 11 18:49:48.647112 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:48.646919 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7ca6df-6cb6cb5765-c86n8" Sep 11 18:49:48.647112 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:48.646969 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7ca6df-6cb6cb5765-c86n8" event={"ID":"0ec34a6b-f340-4743-a761-b5327b9e7516","Type":"ContainerDied","Data":"7047447bfa9db574277d479bc02bd507aaacfada7df24f1f94ab4851bba3d70f"} Sep 11 18:49:48.647112 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:48.647035 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7ca6df-6cb6cb5765-c86n8" event={"ID":"0ec34a6b-f340-4743-a761-b5327b9e7516","Type":"ContainerDied","Data":"da99cbb367adac7c3987b522cafe1e84e8795b224b453232568749b6f222a39e"} Sep 11 18:49:48.647112 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:48.647094 2561 scope.go:117] "RemoveContainer" containerID="7047447bfa9db574277d479bc02bd507aaacfada7df24f1f94ab4851bba3d70f" Sep 11 18:49:48.648617 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:48.648592 2561 generic.go:356] "Generic (PLEG): container finished" podID="0b4dec75-b53b-4e05-82eb-553ac25bee7d" containerID="a0aa8bb0edadd64029c0a08152571f20b5498c55317088b468c82cb13d8f67dc" exitCode=0 Sep 11 18:49:48.648726 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:48.648640 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7ca6df-5c896694c5-qf4zs" Sep 11 18:49:48.648726 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:48.648670 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7ca6df-5c896694c5-qf4zs" event={"ID":"0b4dec75-b53b-4e05-82eb-553ac25bee7d","Type":"ContainerDied","Data":"a0aa8bb0edadd64029c0a08152571f20b5498c55317088b468c82cb13d8f67dc"} Sep 11 18:49:48.648838 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:48.648724 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7ca6df-5c896694c5-qf4zs" event={"ID":"0b4dec75-b53b-4e05-82eb-553ac25bee7d","Type":"ContainerDied","Data":"12214b2008e160b80bcb99cc0ca9bd85b2571d1890d69249a5f707b645922a93"} Sep 11 18:49:48.657699 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:48.657681 2561 scope.go:117] "RemoveContainer" containerID="7047447bfa9db574277d479bc02bd507aaacfada7df24f1f94ab4851bba3d70f" Sep 11 18:49:48.658158 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:49:48.658017 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7047447bfa9db574277d479bc02bd507aaacfada7df24f1f94ab4851bba3d70f\": container with ID starting with 7047447bfa9db574277d479bc02bd507aaacfada7df24f1f94ab4851bba3d70f not found: ID does not exist" containerID="7047447bfa9db574277d479bc02bd507aaacfada7df24f1f94ab4851bba3d70f" Sep 11 18:49:48.658158 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:48.658073 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7047447bfa9db574277d479bc02bd507aaacfada7df24f1f94ab4851bba3d70f"} err="failed to get container status \"7047447bfa9db574277d479bc02bd507aaacfada7df24f1f94ab4851bba3d70f\": rpc error: code = NotFound desc = could not find container \"7047447bfa9db574277d479bc02bd507aaacfada7df24f1f94ab4851bba3d70f\": container with ID starting with 7047447bfa9db574277d479bc02bd507aaacfada7df24f1f94ab4851bba3d70f not found: ID does not exist" Sep 11 18:49:48.658158 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:48.658097 2561 scope.go:117] "RemoveContainer" containerID="a0aa8bb0edadd64029c0a08152571f20b5498c55317088b468c82cb13d8f67dc" Sep 11 18:49:48.666862 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:48.666844 2561 scope.go:117] "RemoveContainer" containerID="a0aa8bb0edadd64029c0a08152571f20b5498c55317088b468c82cb13d8f67dc" Sep 11 18:49:48.667131 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:49:48.667112 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0aa8bb0edadd64029c0a08152571f20b5498c55317088b468c82cb13d8f67dc\": container with ID starting with a0aa8bb0edadd64029c0a08152571f20b5498c55317088b468c82cb13d8f67dc not found: ID does not exist" containerID="a0aa8bb0edadd64029c0a08152571f20b5498c55317088b468c82cb13d8f67dc" Sep 11 18:49:48.667188 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:48.667136 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0aa8bb0edadd64029c0a08152571f20b5498c55317088b468c82cb13d8f67dc"} err="failed to get container status \"a0aa8bb0edadd64029c0a08152571f20b5498c55317088b468c82cb13d8f67dc\": rpc error: code = NotFound desc = could not find container \"a0aa8bb0edadd64029c0a08152571f20b5498c55317088b468c82cb13d8f67dc\": container with ID starting with a0aa8bb0edadd64029c0a08152571f20b5498c55317088b468c82cb13d8f67dc not found: ID does not exist" Sep 11 18:49:48.673964 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:48.673937 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-7ca6df-6cb6cb5765-c86n8"] Sep 11 18:49:48.681046 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:48.681024 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-7ca6df-6cb6cb5765-c86n8"] Sep 11 18:49:48.692374 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:48.692355 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7ca6df-5c896694c5-qf4zs"] Sep 11 18:49:48.697400 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:48.697375 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7ca6df-5c896694c5-qf4zs"] Sep 11 18:49:49.146400 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:49.143891 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b4dec75-b53b-4e05-82eb-553ac25bee7d" path="/var/lib/kubelet/pods/0b4dec75-b53b-4e05-82eb-553ac25bee7d/volumes" Sep 11 18:49:49.146400 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:49:49.144699 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ec34a6b-f340-4743-a761-b5327b9e7516" path="/var/lib/kubelet/pods/0ec34a6b-f340-4743-a761-b5327b9e7516/volumes" Sep 11 18:50:06.128949 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:50:06.128913 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-711a34-6689bb58bb-d58vr"] Sep 11 18:50:06.129524 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:50:06.129127 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-711a34-6689bb58bb-d58vr" podUID="bbe7734f-b586-494a-a1d2-30adae630060" containerName="maas-api" containerID="cri-o://d1bebd9c348a1bb9b7ab451a4dafe27b9f3c379740ede00ce1f989157df4438c" gracePeriod=30 Sep 11 18:50:06.151842 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:50:06.151799 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-711a34-6689bb58bb-d58vr" podUID="bbe7734f-b586-494a-a1d2-30adae630060" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.98:8443/health\": dial tcp 10.132.0.98:8443: connect: connection refused" Sep 11 18:50:06.383987 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:50:06.383912 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-711a34-6689bb58bb-d58vr" Sep 11 18:50:06.470020 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:50:06.469987 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbe7734f-b586-494a-a1d2-30adae630060-maas-api-tls\") pod \"bbe7734f-b586-494a-a1d2-30adae630060\" (UID: \"bbe7734f-b586-494a-a1d2-30adae630060\") " Sep 11 18:50:06.470237 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:50:06.470109 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbe7734f-b586-494a-a1d2-30adae630060-metrics-certs\") pod \"bbe7734f-b586-494a-a1d2-30adae630060\" (UID: \"bbe7734f-b586-494a-a1d2-30adae630060\") " Sep 11 18:50:06.470237 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:50:06.470161 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbe7734f-b586-494a-a1d2-30adae630060-openshift-service-ca\") pod \"bbe7734f-b586-494a-a1d2-30adae630060\" (UID: \"bbe7734f-b586-494a-a1d2-30adae630060\") " Sep 11 18:50:06.470237 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:50:06.470196 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5sfmm\" (UniqueName: \"kubernetes.io/projected/bbe7734f-b586-494a-a1d2-30adae630060-kube-api-access-5sfmm\") pod \"bbe7734f-b586-494a-a1d2-30adae630060\" (UID: \"bbe7734f-b586-494a-a1d2-30adae630060\") " Sep 11 18:50:06.470707 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:50:06.470585 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbe7734f-b586-494a-a1d2-30adae630060-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bbe7734f-b586-494a-a1d2-30adae630060" (UID: "bbe7734f-b586-494a-a1d2-30adae630060"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:50:06.472408 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:50:06.472374 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbe7734f-b586-494a-a1d2-30adae630060-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bbe7734f-b586-494a-a1d2-30adae630060" (UID: "bbe7734f-b586-494a-a1d2-30adae630060"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:50:06.472408 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:50:06.472401 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbe7734f-b586-494a-a1d2-30adae630060-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bbe7734f-b586-494a-a1d2-30adae630060" (UID: "bbe7734f-b586-494a-a1d2-30adae630060"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:50:06.472725 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:50:06.472703 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbe7734f-b586-494a-a1d2-30adae630060-kube-api-access-5sfmm" (OuterVolumeSpecName: "kube-api-access-5sfmm") pod "bbe7734f-b586-494a-a1d2-30adae630060" (UID: "bbe7734f-b586-494a-a1d2-30adae630060"). InnerVolumeSpecName "kube-api-access-5sfmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:50:06.570915 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:50:06.570884 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sfmm\" (UniqueName: \"kubernetes.io/projected/bbe7734f-b586-494a-a1d2-30adae630060-kube-api-access-5sfmm\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:50:06.570915 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:50:06.570911 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbe7734f-b586-494a-a1d2-30adae630060-maas-api-tls\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:50:06.570915 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:50:06.570920 2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbe7734f-b586-494a-a1d2-30adae630060-metrics-certs\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:50:06.571187 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:50:06.570928 2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbe7734f-b586-494a-a1d2-30adae630060-openshift-service-ca\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:50:06.759275 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:50:06.759241 2561 generic.go:356] "Generic (PLEG): container finished" podID="bbe7734f-b586-494a-a1d2-30adae630060" containerID="d1bebd9c348a1bb9b7ab451a4dafe27b9f3c379740ede00ce1f989157df4438c" exitCode=0 Sep 11 18:50:06.759438 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:50:06.759286 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-711a34-6689bb58bb-d58vr" Sep 11 18:50:06.759438 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:50:06.759331 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-711a34-6689bb58bb-d58vr" event={"ID":"bbe7734f-b586-494a-a1d2-30adae630060","Type":"ContainerDied","Data":"d1bebd9c348a1bb9b7ab451a4dafe27b9f3c379740ede00ce1f989157df4438c"} Sep 11 18:50:06.759438 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:50:06.759384 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-711a34-6689bb58bb-d58vr" event={"ID":"bbe7734f-b586-494a-a1d2-30adae630060","Type":"ContainerDied","Data":"190249f9cbfc79f057e7dc19a8619b251f834a2256c54f170ec9c57eee71592c"} Sep 11 18:50:06.759438 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:50:06.759412 2561 scope.go:117] "RemoveContainer" containerID="d1bebd9c348a1bb9b7ab451a4dafe27b9f3c379740ede00ce1f989157df4438c" Sep 11 18:50:06.769396 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:50:06.769376 2561 scope.go:117] "RemoveContainer" containerID="d1bebd9c348a1bb9b7ab451a4dafe27b9f3c379740ede00ce1f989157df4438c" Sep 11 18:50:06.769707 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:50:06.769681 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1bebd9c348a1bb9b7ab451a4dafe27b9f3c379740ede00ce1f989157df4438c\": container with ID starting with d1bebd9c348a1bb9b7ab451a4dafe27b9f3c379740ede00ce1f989157df4438c not found: ID does not exist" containerID="d1bebd9c348a1bb9b7ab451a4dafe27b9f3c379740ede00ce1f989157df4438c" Sep 11 18:50:06.769803 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:50:06.769713 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1bebd9c348a1bb9b7ab451a4dafe27b9f3c379740ede00ce1f989157df4438c"} err="failed to get container status \"d1bebd9c348a1bb9b7ab451a4dafe27b9f3c379740ede00ce1f989157df4438c\": rpc error: code = NotFound desc = could not find container \"d1bebd9c348a1bb9b7ab451a4dafe27b9f3c379740ede00ce1f989157df4438c\": container with ID starting with d1bebd9c348a1bb9b7ab451a4dafe27b9f3c379740ede00ce1f989157df4438c not found: ID does not exist" Sep 11 18:50:06.784357 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:50:06.784329 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-711a34-6689bb58bb-d58vr"] Sep 11 18:50:06.787024 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:50:06.787000 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-711a34-6689bb58bb-d58vr"] Sep 11 18:50:07.141656 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:50:07.141577 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbe7734f-b586-494a-a1d2-30adae630060" path="/var/lib/kubelet/pods/bbe7734f-b586-494a-a1d2-30adae630060/volumes" Sep 11 18:50:19.575873 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:50:19.575846 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dj524_60702ef3-98ef-463b-b877-3a00825353d8/console-operator/1.log" Sep 11 18:50:19.583824 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:50:19.583797 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4knd_42e5ddd8-a96c-45a1-adf9-caa47ee902db/ovn-acl-logging/0.log" Sep 11 18:50:19.587544 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:50:19.587522 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dj524_60702ef3-98ef-463b-b877-3a00825353d8/console-operator/1.log" Sep 11 18:50:19.594780 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:50:19.594760 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4knd_42e5ddd8-a96c-45a1-adf9-caa47ee902db/ovn-acl-logging/0.log" Sep 11 18:52:49.201614 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:52:49.201576 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lsktr"] Sep 11 18:52:49.204276 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:52:49.201861 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lsktr" podUID="0059eaea-499d-499c-98d6-9547fc02722d" containerName="manager" containerID="cri-o://ce74c0a1160ab916eb52557874dc9bb5a904eb6127fd099f33a71743c580205a" gracePeriod=10 Sep 11 18:52:50.677212 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:52:50.677182 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lsktr" Sep 11 18:52:50.736904 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:52:50.736798 2561 generic.go:356] "Generic (PLEG): container finished" podID="0059eaea-499d-499c-98d6-9547fc02722d" containerID="ce74c0a1160ab916eb52557874dc9bb5a904eb6127fd099f33a71743c580205a" exitCode=0 Sep 11 18:52:50.736904 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:52:50.736852 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lsktr" Sep 11 18:52:50.737220 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:52:50.736903 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lsktr" event={"ID":"0059eaea-499d-499c-98d6-9547fc02722d","Type":"ContainerDied","Data":"ce74c0a1160ab916eb52557874dc9bb5a904eb6127fd099f33a71743c580205a"} Sep 11 18:52:50.737220 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:52:50.737018 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lsktr" event={"ID":"0059eaea-499d-499c-98d6-9547fc02722d","Type":"ContainerDied","Data":"e5fbcf9437aa5c3fbe774a50507db4b4370bfc773475e5612609582bc67c70fa"} Sep 11 18:52:50.737220 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:52:50.737121 2561 scope.go:117] "RemoveContainer" containerID="ce74c0a1160ab916eb52557874dc9bb5a904eb6127fd099f33a71743c580205a" Sep 11 18:52:50.747245 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:52:50.747219 2561 scope.go:117] "RemoveContainer" containerID="5f3653ee21cd4951b8d64ae6e2780bd1b6d3209aa53a3fc3f2a62995319f9f39" Sep 11 18:52:50.756806 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:52:50.756785 2561 scope.go:117] "RemoveContainer" containerID="ce74c0a1160ab916eb52557874dc9bb5a904eb6127fd099f33a71743c580205a" Sep 11 18:52:50.757151 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:52:50.757132 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce74c0a1160ab916eb52557874dc9bb5a904eb6127fd099f33a71743c580205a\": container with ID starting with ce74c0a1160ab916eb52557874dc9bb5a904eb6127fd099f33a71743c580205a not found: ID does not exist" containerID="ce74c0a1160ab916eb52557874dc9bb5a904eb6127fd099f33a71743c580205a" Sep 11 18:52:50.757251 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:52:50.757158 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce74c0a1160ab916eb52557874dc9bb5a904eb6127fd099f33a71743c580205a"} err="failed to get container status \"ce74c0a1160ab916eb52557874dc9bb5a904eb6127fd099f33a71743c580205a\": rpc error: code = NotFound desc = could not find container \"ce74c0a1160ab916eb52557874dc9bb5a904eb6127fd099f33a71743c580205a\": container with ID starting with ce74c0a1160ab916eb52557874dc9bb5a904eb6127fd099f33a71743c580205a not found: ID does not exist" Sep 11 18:52:50.757251 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:52:50.757177 2561 scope.go:117] "RemoveContainer" containerID="5f3653ee21cd4951b8d64ae6e2780bd1b6d3209aa53a3fc3f2a62995319f9f39" Sep 11 18:52:50.757461 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:52:50.757440 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f3653ee21cd4951b8d64ae6e2780bd1b6d3209aa53a3fc3f2a62995319f9f39\": container with ID starting with 5f3653ee21cd4951b8d64ae6e2780bd1b6d3209aa53a3fc3f2a62995319f9f39 not found: ID does not exist" containerID="5f3653ee21cd4951b8d64ae6e2780bd1b6d3209aa53a3fc3f2a62995319f9f39" Sep 11 18:52:50.757524 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:52:50.757466 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f3653ee21cd4951b8d64ae6e2780bd1b6d3209aa53a3fc3f2a62995319f9f39"} err="failed to get container status \"5f3653ee21cd4951b8d64ae6e2780bd1b6d3209aa53a3fc3f2a62995319f9f39\": rpc error: code = NotFound desc = could not find container \"5f3653ee21cd4951b8d64ae6e2780bd1b6d3209aa53a3fc3f2a62995319f9f39\": container with ID starting with 5f3653ee21cd4951b8d64ae6e2780bd1b6d3209aa53a3fc3f2a62995319f9f39 not found: ID does not exist" Sep 11 18:52:50.824245 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:52:50.824212 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cbvt8\" (UniqueName: \"kubernetes.io/projected/0059eaea-499d-499c-98d6-9547fc02722d-kube-api-access-cbvt8\") pod \"0059eaea-499d-499c-98d6-9547fc02722d\" (UID: \"0059eaea-499d-499c-98d6-9547fc02722d\") " Sep 11 18:52:50.824395 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:52:50.824252 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0059eaea-499d-499c-98d6-9547fc02722d-extensions-socket-volume\") pod \"0059eaea-499d-499c-98d6-9547fc02722d\" (UID: \"0059eaea-499d-499c-98d6-9547fc02722d\") " Sep 11 18:52:50.824745 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:52:50.824719 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0059eaea-499d-499c-98d6-9547fc02722d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "0059eaea-499d-499c-98d6-9547fc02722d" (UID: "0059eaea-499d-499c-98d6-9547fc02722d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:52:50.826436 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:52:50.826405 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0059eaea-499d-499c-98d6-9547fc02722d-kube-api-access-cbvt8" (OuterVolumeSpecName: "kube-api-access-cbvt8") pod "0059eaea-499d-499c-98d6-9547fc02722d" (UID: "0059eaea-499d-499c-98d6-9547fc02722d"). InnerVolumeSpecName "kube-api-access-cbvt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:52:50.925322 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:52:50.925269 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbvt8\" (UniqueName: \"kubernetes.io/projected/0059eaea-499d-499c-98d6-9547fc02722d-kube-api-access-cbvt8\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:52:50.925322 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:52:50.925317 2561 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0059eaea-499d-499c-98d6-9547fc02722d-extensions-socket-volume\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:52:51.063903 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:52:51.063869 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lsktr"] Sep 11 18:52:51.071481 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:52:51.071452 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lsktr"] Sep 11 18:52:51.139969 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:52:51.139937 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0059eaea-499d-499c-98d6-9547fc02722d" path="/var/lib/kubelet/pods/0059eaea-499d-499c-98d6-9547fc02722d/volumes" Sep 11 18:53:49.868659 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:53:49.868619 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-r68m2"] Sep 11 18:53:49.869202 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:53:49.869002 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0059eaea-499d-499c-98d6-9547fc02722d" containerName="manager" Sep 11 18:53:49.869202 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:53:49.869014 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0059eaea-499d-499c-98d6-9547fc02722d" containerName="manager" Sep 11 18:53:49.869202 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:53:49.869032 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b4dec75-b53b-4e05-82eb-553ac25bee7d" containerName="maas-api" Sep 11 18:53:49.869202 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:53:49.869038 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b4dec75-b53b-4e05-82eb-553ac25bee7d" containerName="maas-api" Sep 11 18:53:49.869202 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:53:49.869046 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbe7734f-b586-494a-a1d2-30adae630060" containerName="maas-api" Sep 11 18:53:49.869202 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:53:49.869068 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbe7734f-b586-494a-a1d2-30adae630060" containerName="maas-api" Sep 11 18:53:49.869202 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:53:49.869083 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f" containerName="istio-proxy" Sep 11 18:53:49.869202 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:53:49.869093 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f" containerName="istio-proxy" Sep 11 18:53:49.869202 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:53:49.869103 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e80d9a66-2f1b-4f88-b1c1-3cd31958d700" containerName="maas-api" Sep 11 18:53:49.869202 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:53:49.869110 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e80d9a66-2f1b-4f88-b1c1-3cd31958d700" containerName="maas-api" Sep 11 18:53:49.869202 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:53:49.869120 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ec34a6b-f340-4743-a761-b5327b9e7516" containerName="payload-processing" Sep 11 18:53:49.869202 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:53:49.869125 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ec34a6b-f340-4743-a761-b5327b9e7516" containerName="payload-processing" Sep 11 18:53:49.869202 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:53:49.869185 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b4dec75-b53b-4e05-82eb-553ac25bee7d" containerName="maas-api" Sep 11 18:53:49.869202 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:53:49.869191 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbe7734f-b586-494a-a1d2-30adae630060" containerName="maas-api" Sep 11 18:53:49.869202 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:53:49.869199 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f" containerName="istio-proxy" Sep 11 18:53:49.869202 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:53:49.869204 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e80d9a66-2f1b-4f88-b1c1-3cd31958d700" containerName="maas-api" Sep 11 18:53:49.869202 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:53:49.869209 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0059eaea-499d-499c-98d6-9547fc02722d" containerName="manager" Sep 11 18:53:49.869202 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:53:49.869216 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0059eaea-499d-499c-98d6-9547fc02722d" containerName="manager" Sep 11 18:53:49.870106 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:53:49.869221 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ec34a6b-f340-4743-a761-b5327b9e7516" containerName="payload-processing" Sep 11 18:53:49.870106 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:53:49.869227 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f" containerName="istio-proxy" Sep 11 18:53:49.873155 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:53:49.873126 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-r68m2" Sep 11 18:53:49.875851 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:53:49.875829 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-j46f2\"" Sep 11 18:53:49.887190 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:53:49.887168 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-r68m2"] Sep 11 18:53:49.973171 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:53:49.973140 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2pb9\" (UniqueName: \"kubernetes.io/projected/51293999-b24d-48cc-8b7c-ad10331ab72e-kube-api-access-c2pb9\") pod \"kuadrant-operator-controller-manager-5769f99d9f-r68m2\" (UID: \"51293999-b24d-48cc-8b7c-ad10331ab72e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-r68m2" Sep 11 18:53:49.973331 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:53:49.973257 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/51293999-b24d-48cc-8b7c-ad10331ab72e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-r68m2\" (UID: \"51293999-b24d-48cc-8b7c-ad10331ab72e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-r68m2" Sep 11 18:53:50.074619 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:53:50.074589 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2pb9\" (UniqueName: \"kubernetes.io/projected/51293999-b24d-48cc-8b7c-ad10331ab72e-kube-api-access-c2pb9\") pod \"kuadrant-operator-controller-manager-5769f99d9f-r68m2\" (UID: \"51293999-b24d-48cc-8b7c-ad10331ab72e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-r68m2" Sep 11 18:53:50.074766 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:53:50.074670 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/51293999-b24d-48cc-8b7c-ad10331ab72e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-r68m2\" (UID: \"51293999-b24d-48cc-8b7c-ad10331ab72e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-r68m2" Sep 11 18:53:50.075013 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:53:50.074997 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/51293999-b24d-48cc-8b7c-ad10331ab72e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-r68m2\" (UID: \"51293999-b24d-48cc-8b7c-ad10331ab72e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-r68m2" Sep 11 18:53:50.083311 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:53:50.083288 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2pb9\" (UniqueName: \"kubernetes.io/projected/51293999-b24d-48cc-8b7c-ad10331ab72e-kube-api-access-c2pb9\") pod \"kuadrant-operator-controller-manager-5769f99d9f-r68m2\" (UID: \"51293999-b24d-48cc-8b7c-ad10331ab72e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-r68m2" Sep 11 18:53:50.185616 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:53:50.185590 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-r68m2" Sep 11 18:53:50.467962 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:53:50.467930 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-r68m2"] Sep 11 18:53:50.475202 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:53:50.475167 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51293999_b24d_48cc_8b7c_ad10331ab72e.slice/crio-70778bfcf10fee477a5d55a066bfecd7d6200b85d0a8f5cdf59d5757b3a255d1 WatchSource:0}: Error finding container 70778bfcf10fee477a5d55a066bfecd7d6200b85d0a8f5cdf59d5757b3a255d1: Status 404 returned error can't find the container with id 70778bfcf10fee477a5d55a066bfecd7d6200b85d0a8f5cdf59d5757b3a255d1 Sep 11 18:53:51.078558 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:53:51.078517 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-r68m2" event={"ID":"51293999-b24d-48cc-8b7c-ad10331ab72e","Type":"ContainerStarted","Data":"ce451de672acfce5e451316033fb2c93be5e9ed2e1fe5b79d25d30a32b8f5250"} Sep 11 18:53:51.078993 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:53:51.078575 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-r68m2" event={"ID":"51293999-b24d-48cc-8b7c-ad10331ab72e","Type":"ContainerStarted","Data":"70778bfcf10fee477a5d55a066bfecd7d6200b85d0a8f5cdf59d5757b3a255d1"} Sep 11 18:53:51.078993 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:53:51.078663 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-r68m2" Sep 11 18:53:51.099166 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:53:51.099107 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-r68m2" podStartSLOduration=2.099088848 podStartE2EDuration="2.099088848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:53:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:53:51.097291583 +0000 UTC m=+2312.594309336" watchObservedRunningTime="2026-09-11 18:53:51.099088848 +0000 UTC m=+2312.596106605" Sep 11 18:54:02.086434 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:54:02.086398 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-r68m2" Sep 11 18:55:12.804356 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:12.804315 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6b5bd7959c-txpb7"] Sep 11 18:55:12.805077 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:12.805034 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f" containerName="istio-proxy" Sep 11 18:55:12.805261 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:12.805231 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2756ee4d-8f93-4182-9c1e-2a4b56fa8a1f" containerName="istio-proxy" Sep 11 18:55:12.805366 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:12.805299 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0059eaea-499d-499c-98d6-9547fc02722d" containerName="manager" Sep 11 18:55:12.805366 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:12.805311 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0059eaea-499d-499c-98d6-9547fc02722d" containerName="manager" Sep 11 18:55:12.808647 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:12.808623 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b5bd7959c-txpb7" Sep 11 18:55:12.811665 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:12.811646 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 11 18:55:12.811788 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:12.811769 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 11 18:55:12.811861 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:12.811801 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 11 18:55:12.811861 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:12.811837 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 18:55:12.811971 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:12.811801 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-c9xd9\"" Sep 11 18:55:12.820646 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:12.820622 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b5bd7959c-txpb7"] Sep 11 18:55:12.949560 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:12.949521 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnfpq\" (UniqueName: \"kubernetes.io/projected/2441eb50-2c4d-419c-9d0c-696296aba6ed-kube-api-access-mnfpq\") pod \"maas-api-6b5bd7959c-txpb7\" (UID: \"2441eb50-2c4d-419c-9d0c-696296aba6ed\") " pod="odh-ai-gateway-infra/maas-api-6b5bd7959c-txpb7" Sep 11 18:55:12.949732 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:12.949590 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2441eb50-2c4d-419c-9d0c-696296aba6ed-openshift-service-ca\") pod \"maas-api-6b5bd7959c-txpb7\" (UID: \"2441eb50-2c4d-419c-9d0c-696296aba6ed\") " pod="odh-ai-gateway-infra/maas-api-6b5bd7959c-txpb7" Sep 11 18:55:12.949732 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:12.949661 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2441eb50-2c4d-419c-9d0c-696296aba6ed-maas-api-tls\") pod \"maas-api-6b5bd7959c-txpb7\" (UID: \"2441eb50-2c4d-419c-9d0c-696296aba6ed\") " pod="odh-ai-gateway-infra/maas-api-6b5bd7959c-txpb7" Sep 11 18:55:12.949732 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:12.949695 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2441eb50-2c4d-419c-9d0c-696296aba6ed-metrics-certs\") pod \"maas-api-6b5bd7959c-txpb7\" (UID: \"2441eb50-2c4d-419c-9d0c-696296aba6ed\") " pod="odh-ai-gateway-infra/maas-api-6b5bd7959c-txpb7" Sep 11 18:55:13.050837 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:13.050795 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2441eb50-2c4d-419c-9d0c-696296aba6ed-openshift-service-ca\") pod \"maas-api-6b5bd7959c-txpb7\" (UID: \"2441eb50-2c4d-419c-9d0c-696296aba6ed\") " pod="odh-ai-gateway-infra/maas-api-6b5bd7959c-txpb7" Sep 11 18:55:13.051005 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:13.050886 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2441eb50-2c4d-419c-9d0c-696296aba6ed-maas-api-tls\") pod \"maas-api-6b5bd7959c-txpb7\" (UID: \"2441eb50-2c4d-419c-9d0c-696296aba6ed\") " pod="odh-ai-gateway-infra/maas-api-6b5bd7959c-txpb7" Sep 11 18:55:13.051005 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:13.050930 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2441eb50-2c4d-419c-9d0c-696296aba6ed-metrics-certs\") pod \"maas-api-6b5bd7959c-txpb7\" (UID: \"2441eb50-2c4d-419c-9d0c-696296aba6ed\") " pod="odh-ai-gateway-infra/maas-api-6b5bd7959c-txpb7" Sep 11 18:55:13.051156 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:13.050994 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnfpq\" (UniqueName: \"kubernetes.io/projected/2441eb50-2c4d-419c-9d0c-696296aba6ed-kube-api-access-mnfpq\") pod \"maas-api-6b5bd7959c-txpb7\" (UID: \"2441eb50-2c4d-419c-9d0c-696296aba6ed\") " pod="odh-ai-gateway-infra/maas-api-6b5bd7959c-txpb7" Sep 11 18:55:13.051636 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:13.051608 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2441eb50-2c4d-419c-9d0c-696296aba6ed-openshift-service-ca\") pod \"maas-api-6b5bd7959c-txpb7\" (UID: \"2441eb50-2c4d-419c-9d0c-696296aba6ed\") " pod="odh-ai-gateway-infra/maas-api-6b5bd7959c-txpb7" Sep 11 18:55:13.053517 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:13.053493 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2441eb50-2c4d-419c-9d0c-696296aba6ed-metrics-certs\") pod \"maas-api-6b5bd7959c-txpb7\" (UID: \"2441eb50-2c4d-419c-9d0c-696296aba6ed\") " pod="odh-ai-gateway-infra/maas-api-6b5bd7959c-txpb7" Sep 11 18:55:13.053633 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:13.053527 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2441eb50-2c4d-419c-9d0c-696296aba6ed-maas-api-tls\") pod \"maas-api-6b5bd7959c-txpb7\" (UID: \"2441eb50-2c4d-419c-9d0c-696296aba6ed\") " pod="odh-ai-gateway-infra/maas-api-6b5bd7959c-txpb7" Sep 11 18:55:13.059902 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:13.059836 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnfpq\" (UniqueName: \"kubernetes.io/projected/2441eb50-2c4d-419c-9d0c-696296aba6ed-kube-api-access-mnfpq\") pod \"maas-api-6b5bd7959c-txpb7\" (UID: \"2441eb50-2c4d-419c-9d0c-696296aba6ed\") " pod="odh-ai-gateway-infra/maas-api-6b5bd7959c-txpb7" Sep 11 18:55:13.120922 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:13.120886 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b5bd7959c-txpb7" Sep 11 18:55:13.401586 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:13.401554 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b5bd7959c-txpb7"] Sep 11 18:55:13.403540 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:55:13.403509 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2441eb50_2c4d_419c_9d0c_696296aba6ed.slice/crio-b2c364a38aece676da561e29d681f778f06a98ecda63af6001f446852999b82e WatchSource:0}: Error finding container b2c364a38aece676da561e29d681f778f06a98ecda63af6001f446852999b82e: Status 404 returned error can't find the container with id b2c364a38aece676da561e29d681f778f06a98ecda63af6001f446852999b82e Sep 11 18:55:13.404975 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:13.404957 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 18:55:13.564413 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:13.564377 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b5bd7959c-txpb7" event={"ID":"2441eb50-2c4d-419c-9d0c-696296aba6ed","Type":"ContainerStarted","Data":"b2c364a38aece676da561e29d681f778f06a98ecda63af6001f446852999b82e"} Sep 11 18:55:14.572342 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:14.572298 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b5bd7959c-txpb7" event={"ID":"2441eb50-2c4d-419c-9d0c-696296aba6ed","Type":"ContainerStarted","Data":"4843cc0b2d922e74d0b987458aa5eb28ed1df1a2457d2a3e09db5e5e81ffd226"} Sep 11 18:55:14.572763 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:14.572374 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6b5bd7959c-txpb7" Sep 11 18:55:14.593974 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:14.593913 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6b5bd7959c-txpb7" podStartSLOduration=2.170364411 podStartE2EDuration="2.593892271s" totalImagesPullingTime="423.52786ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:55:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:55:14.589727953 +0000 UTC m=+2396.086745706" watchObservedRunningTime="2026-09-11 18:55:14.593892271 +0000 UTC m=+2396.090910026" Sep 11 18:55:19.617199 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:19.617169 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dj524_60702ef3-98ef-463b-b877-3a00825353d8/console-operator/1.log" Sep 11 18:55:19.628394 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:19.628368 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4knd_42e5ddd8-a96c-45a1-adf9-caa47ee902db/ovn-acl-logging/0.log" Sep 11 18:55:19.632092 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:19.632049 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dj524_60702ef3-98ef-463b-b877-3a00825353d8/console-operator/1.log" Sep 11 18:55:19.639452 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:19.639427 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4knd_42e5ddd8-a96c-45a1-adf9-caa47ee902db/ovn-acl-logging/0.log" Sep 11 18:55:20.583859 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:20.583832 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6b5bd7959c-txpb7" Sep 11 18:55:20.853651 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:20.853553 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-798f7d6bf7-m5mt6"] Sep 11 18:55:20.859685 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:20.859645 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-798f7d6bf7-m5mt6" Sep 11 18:55:20.866374 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:20.866340 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-798f7d6bf7-m5mt6"] Sep 11 18:55:20.916258 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:20.916220 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e64a44e-32c1-4e5b-afca-76bfe67ce2dd-openshift-service-ca\") pod \"maas-api-798f7d6bf7-m5mt6\" (UID: \"7e64a44e-32c1-4e5b-afca-76bfe67ce2dd\") " pod="odh-ai-gateway-infra/maas-api-798f7d6bf7-m5mt6" Sep 11 18:55:20.916258 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:20.916261 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg7bp\" (UniqueName: \"kubernetes.io/projected/7e64a44e-32c1-4e5b-afca-76bfe67ce2dd-kube-api-access-cg7bp\") pod \"maas-api-798f7d6bf7-m5mt6\" (UID: \"7e64a44e-32c1-4e5b-afca-76bfe67ce2dd\") " pod="odh-ai-gateway-infra/maas-api-798f7d6bf7-m5mt6" Sep 11 18:55:20.916477 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:20.916304 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e64a44e-32c1-4e5b-afca-76bfe67ce2dd-maas-api-tls\") pod \"maas-api-798f7d6bf7-m5mt6\" (UID: \"7e64a44e-32c1-4e5b-afca-76bfe67ce2dd\") " pod="odh-ai-gateway-infra/maas-api-798f7d6bf7-m5mt6" Sep 11 18:55:20.916477 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:20.916366 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e64a44e-32c1-4e5b-afca-76bfe67ce2dd-metrics-certs\") pod \"maas-api-798f7d6bf7-m5mt6\" (UID: \"7e64a44e-32c1-4e5b-afca-76bfe67ce2dd\") " pod="odh-ai-gateway-infra/maas-api-798f7d6bf7-m5mt6" Sep 11 18:55:21.017166 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:21.017127 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e64a44e-32c1-4e5b-afca-76bfe67ce2dd-maas-api-tls\") pod \"maas-api-798f7d6bf7-m5mt6\" (UID: \"7e64a44e-32c1-4e5b-afca-76bfe67ce2dd\") " pod="odh-ai-gateway-infra/maas-api-798f7d6bf7-m5mt6" Sep 11 18:55:21.017345 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:21.017221 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e64a44e-32c1-4e5b-afca-76bfe67ce2dd-metrics-certs\") pod \"maas-api-798f7d6bf7-m5mt6\" (UID: \"7e64a44e-32c1-4e5b-afca-76bfe67ce2dd\") " pod="odh-ai-gateway-infra/maas-api-798f7d6bf7-m5mt6" Sep 11 18:55:21.017345 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:21.017271 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e64a44e-32c1-4e5b-afca-76bfe67ce2dd-openshift-service-ca\") pod \"maas-api-798f7d6bf7-m5mt6\" (UID: \"7e64a44e-32c1-4e5b-afca-76bfe67ce2dd\") " pod="odh-ai-gateway-infra/maas-api-798f7d6bf7-m5mt6" Sep 11 18:55:21.017443 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:21.017420 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg7bp\" (UniqueName: \"kubernetes.io/projected/7e64a44e-32c1-4e5b-afca-76bfe67ce2dd-kube-api-access-cg7bp\") pod \"maas-api-798f7d6bf7-m5mt6\" (UID: \"7e64a44e-32c1-4e5b-afca-76bfe67ce2dd\") " pod="odh-ai-gateway-infra/maas-api-798f7d6bf7-m5mt6" Sep 11 18:55:21.018073 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:21.018031 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e64a44e-32c1-4e5b-afca-76bfe67ce2dd-openshift-service-ca\") pod \"maas-api-798f7d6bf7-m5mt6\" (UID: \"7e64a44e-32c1-4e5b-afca-76bfe67ce2dd\") " pod="odh-ai-gateway-infra/maas-api-798f7d6bf7-m5mt6" Sep 11 18:55:21.019774 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:21.019754 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e64a44e-32c1-4e5b-afca-76bfe67ce2dd-metrics-certs\") pod \"maas-api-798f7d6bf7-m5mt6\" (UID: \"7e64a44e-32c1-4e5b-afca-76bfe67ce2dd\") " pod="odh-ai-gateway-infra/maas-api-798f7d6bf7-m5mt6" Sep 11 18:55:21.020170 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:21.020154 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e64a44e-32c1-4e5b-afca-76bfe67ce2dd-maas-api-tls\") pod \"maas-api-798f7d6bf7-m5mt6\" (UID: \"7e64a44e-32c1-4e5b-afca-76bfe67ce2dd\") " pod="odh-ai-gateway-infra/maas-api-798f7d6bf7-m5mt6" Sep 11 18:55:21.026251 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:21.026220 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg7bp\" (UniqueName: \"kubernetes.io/projected/7e64a44e-32c1-4e5b-afca-76bfe67ce2dd-kube-api-access-cg7bp\") pod \"maas-api-798f7d6bf7-m5mt6\" (UID: \"7e64a44e-32c1-4e5b-afca-76bfe67ce2dd\") " pod="odh-ai-gateway-infra/maas-api-798f7d6bf7-m5mt6" Sep 11 18:55:21.173303 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:21.173264 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-798f7d6bf7-m5mt6" Sep 11 18:55:21.457658 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:21.457625 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-798f7d6bf7-m5mt6"] Sep 11 18:55:21.458877 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:55:21.458853 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e64a44e_32c1_4e5b_afca_76bfe67ce2dd.slice/crio-322b45c2fcc0b4ec39e62410a85884369b8b1242c8f1dbd0495cd6a2864c98eb WatchSource:0}: Error finding container 322b45c2fcc0b4ec39e62410a85884369b8b1242c8f1dbd0495cd6a2864c98eb: Status 404 returned error can't find the container with id 322b45c2fcc0b4ec39e62410a85884369b8b1242c8f1dbd0495cd6a2864c98eb Sep 11 18:55:21.612888 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:21.612848 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-798f7d6bf7-m5mt6" event={"ID":"7e64a44e-32c1-4e5b-afca-76bfe67ce2dd","Type":"ContainerStarted","Data":"322b45c2fcc0b4ec39e62410a85884369b8b1242c8f1dbd0495cd6a2864c98eb"} Sep 11 18:55:22.620458 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:22.620416 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-798f7d6bf7-m5mt6" event={"ID":"7e64a44e-32c1-4e5b-afca-76bfe67ce2dd","Type":"ContainerStarted","Data":"f4342827accc73b2e4f153e39e0a2820725fa47db2cba6d5dee7af51f8f43591"} Sep 11 18:55:22.620900 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:22.620590 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-798f7d6bf7-m5mt6" Sep 11 18:55:22.640126 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:22.640075 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-798f7d6bf7-m5mt6" podStartSLOduration=2.124988362 podStartE2EDuration="2.640044778s" totalImagesPullingTime="515.056416ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:55:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:55:22.637415237 +0000 UTC m=+2404.134432991" watchObservedRunningTime="2026-09-11 18:55:22.640044778 +0000 UTC m=+2404.137062567" Sep 11 18:55:28.631738 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:28.631710 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-798f7d6bf7-m5mt6" Sep 11 18:55:28.694594 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:28.694554 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b5bd7959c-txpb7"] Sep 11 18:55:28.695114 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:28.695076 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6b5bd7959c-txpb7" podUID="2441eb50-2c4d-419c-9d0c-696296aba6ed" containerName="maas-api" containerID="cri-o://4843cc0b2d922e74d0b987458aa5eb28ed1df1a2457d2a3e09db5e5e81ffd226" gracePeriod=30 Sep 11 18:55:28.946400 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:28.946375 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b5bd7959c-txpb7" Sep 11 18:55:28.997456 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:28.997414 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2441eb50-2c4d-419c-9d0c-696296aba6ed-openshift-service-ca\") pod \"2441eb50-2c4d-419c-9d0c-696296aba6ed\" (UID: \"2441eb50-2c4d-419c-9d0c-696296aba6ed\") " Sep 11 18:55:28.997634 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:28.997601 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2441eb50-2c4d-419c-9d0c-696296aba6ed-maas-api-tls\") pod \"2441eb50-2c4d-419c-9d0c-696296aba6ed\" (UID: \"2441eb50-2c4d-419c-9d0c-696296aba6ed\") " Sep 11 18:55:28.997634 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:28.997619 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2441eb50-2c4d-419c-9d0c-696296aba6ed-metrics-certs\") pod \"2441eb50-2c4d-419c-9d0c-696296aba6ed\" (UID: \"2441eb50-2c4d-419c-9d0c-696296aba6ed\") " Sep 11 18:55:28.997792 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:28.997651 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mnfpq\" (UniqueName: \"kubernetes.io/projected/2441eb50-2c4d-419c-9d0c-696296aba6ed-kube-api-access-mnfpq\") pod \"2441eb50-2c4d-419c-9d0c-696296aba6ed\" (UID: \"2441eb50-2c4d-419c-9d0c-696296aba6ed\") " Sep 11 18:55:28.997880 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:28.997838 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2441eb50-2c4d-419c-9d0c-696296aba6ed-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2441eb50-2c4d-419c-9d0c-696296aba6ed" (UID: "2441eb50-2c4d-419c-9d0c-696296aba6ed"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:55:28.998100 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:28.998040 2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2441eb50-2c4d-419c-9d0c-696296aba6ed-openshift-service-ca\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:55:28.999796 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:28.999768 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2441eb50-2c4d-419c-9d0c-696296aba6ed-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2441eb50-2c4d-419c-9d0c-696296aba6ed" (UID: "2441eb50-2c4d-419c-9d0c-696296aba6ed"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:55:28.999908 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:28.999800 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2441eb50-2c4d-419c-9d0c-696296aba6ed-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2441eb50-2c4d-419c-9d0c-696296aba6ed" (UID: "2441eb50-2c4d-419c-9d0c-696296aba6ed"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:55:28.999908 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:28.999832 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2441eb50-2c4d-419c-9d0c-696296aba6ed-kube-api-access-mnfpq" (OuterVolumeSpecName: "kube-api-access-mnfpq") pod "2441eb50-2c4d-419c-9d0c-696296aba6ed" (UID: "2441eb50-2c4d-419c-9d0c-696296aba6ed"). InnerVolumeSpecName "kube-api-access-mnfpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:55:29.099503 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:29.099469 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2441eb50-2c4d-419c-9d0c-696296aba6ed-maas-api-tls\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:55:29.099503 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:29.099501 2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2441eb50-2c4d-419c-9d0c-696296aba6ed-metrics-certs\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:55:29.099503 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:29.099510 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnfpq\" (UniqueName: \"kubernetes.io/projected/2441eb50-2c4d-419c-9d0c-696296aba6ed-kube-api-access-mnfpq\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 18:55:29.663340 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:29.663305 2561 generic.go:356] "Generic (PLEG): container finished" podID="2441eb50-2c4d-419c-9d0c-696296aba6ed" containerID="4843cc0b2d922e74d0b987458aa5eb28ed1df1a2457d2a3e09db5e5e81ffd226" exitCode=0 Sep 11 18:55:29.663764 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:29.663358 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b5bd7959c-txpb7" Sep 11 18:55:29.663764 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:29.663374 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b5bd7959c-txpb7" event={"ID":"2441eb50-2c4d-419c-9d0c-696296aba6ed","Type":"ContainerDied","Data":"4843cc0b2d922e74d0b987458aa5eb28ed1df1a2457d2a3e09db5e5e81ffd226"} Sep 11 18:55:29.663764 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:29.663416 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b5bd7959c-txpb7" event={"ID":"2441eb50-2c4d-419c-9d0c-696296aba6ed","Type":"ContainerDied","Data":"b2c364a38aece676da561e29d681f778f06a98ecda63af6001f446852999b82e"} Sep 11 18:55:29.663764 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:29.663437 2561 scope.go:117] "RemoveContainer" containerID="4843cc0b2d922e74d0b987458aa5eb28ed1df1a2457d2a3e09db5e5e81ffd226" Sep 11 18:55:29.672891 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:29.672864 2561 scope.go:117] "RemoveContainer" containerID="4843cc0b2d922e74d0b987458aa5eb28ed1df1a2457d2a3e09db5e5e81ffd226" Sep 11 18:55:29.673199 ip-10-0-138-128 kubenswrapper[2561]: E0911 18:55:29.673179 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4843cc0b2d922e74d0b987458aa5eb28ed1df1a2457d2a3e09db5e5e81ffd226\": container with ID starting with 4843cc0b2d922e74d0b987458aa5eb28ed1df1a2457d2a3e09db5e5e81ffd226 not found: ID does not exist" containerID="4843cc0b2d922e74d0b987458aa5eb28ed1df1a2457d2a3e09db5e5e81ffd226" Sep 11 18:55:29.673281 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:29.673208 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4843cc0b2d922e74d0b987458aa5eb28ed1df1a2457d2a3e09db5e5e81ffd226"} err="failed to get container status \"4843cc0b2d922e74d0b987458aa5eb28ed1df1a2457d2a3e09db5e5e81ffd226\": rpc error: code = NotFound desc = could not find container \"4843cc0b2d922e74d0b987458aa5eb28ed1df1a2457d2a3e09db5e5e81ffd226\": container with ID starting with 4843cc0b2d922e74d0b987458aa5eb28ed1df1a2457d2a3e09db5e5e81ffd226 not found: ID does not exist" Sep 11 18:55:29.688787 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:29.688748 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b5bd7959c-txpb7"] Sep 11 18:55:29.697198 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:29.697173 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b5bd7959c-txpb7"] Sep 11 18:55:31.139190 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:31.139158 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2441eb50-2c4d-419c-9d0c-696296aba6ed" path="/var/lib/kubelet/pods/2441eb50-2c4d-419c-9d0c-696296aba6ed/volumes" Sep 11 18:55:46.680632 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:46.680594 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5cc6c58786-qn445_6923f940-4731-40d1-be20-530994fb7394/authorino/0.log" Sep 11 18:55:51.925976 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:51.925941 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-rfphm_1329cb52-610f-4b0f-8b8b-102288987992/manager/0.log" Sep 11 18:55:52.048472 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:52.048424 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-tqgww_052c1dcf-bad7-47b8-b5da-8a1ef2459f46/manager/0.log" Sep 11 18:55:52.273454 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:52.273371 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-l6vbm_b9832d55-a490-4d4e-8b98-3b4f676f42e9/server/0.log" Sep 11 18:55:52.392708 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:52.392678 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-mnwhs_bf80fc01-7343-4f52-a954-dcf935983c7c/manager/0.log" Sep 11 18:55:52.757841 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:52.757809 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76cff9b6b6-rg86p_a5a6a746-6467-4629-b647-250c51a1e91f/copy-manifests/0.log" Sep 11 18:55:52.765345 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:52.765315 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76cff9b6b6-rg86p_a5a6a746-6467-4629-b647-250c51a1e91f/manager/0.log" Sep 11 18:55:53.118924 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:53.118835 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-798f7d6bf7-m5mt6_7e64a44e-32c1-4e5b-afca-76bfe67ce2dd/maas-api/0.log" Sep 11 18:55:53.336705 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:53.336677 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29819190-q9l4g_6c688bfb-48db-4bc1-839a-ee5f08c2994b/cleanup/0.log" Sep 11 18:55:54.305604 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:54.305572 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5cc6c58786-qn445_6923f940-4731-40d1-be20-530994fb7394/authorino/0.log" Sep 11 18:55:54.420542 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:54.420508 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-dvxrr_dc27b87c-4bf6-49de-9a7f-a74ee17929f3/manager/0.log" Sep 11 18:55:54.777986 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:54.777953 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-r68m2_51293999-b24d-48cc-8b7c-ad10331ab72e/manager/0.log" Sep 11 18:55:55.006542 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:55.006508 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-79pgz_1e6be196-1d2f-4255-a612-151457495a5f/manager/0.log" Sep 11 18:55:55.350391 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:55.350358 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7_61673abc-9544-4cf5-aad1-4714f9aacabc/istio-proxy/0.log" Sep 11 18:55:55.707637 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:55.707606 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-67fd44999f-t577s_7c271705-ae8e-4715-bb87-d176e6e0ed07/kube-auth-proxy/0.log" Sep 11 18:55:55.829307 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:55.829276 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-cbvsl_09816140-ebbc-4f2d-9e1a-a11e86d773a4/istio-proxy/1.log" Sep 11 18:55:55.946639 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:55.946608 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-ldlb4_cf29bed6-97b3-4988-aa11-ffebe83a653a/payload-pre-processing/0.log" Sep 11 18:55:56.520443 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:56.520415 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-mh9n8_6f4905e1-a9c3-42bd-9205-5c0d9c163e7c/main/0.log" Sep 11 18:55:56.741504 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:56.741477 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-v9pzq_07e2555f-0967-4633-bc02-ad8262d1e749/main/0.log" Sep 11 18:55:57.080218 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:55:57.080190 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-4r97j_5cacdfa6-534e-4981-8189-f5fa8d0e2c97/main/0.log" Sep 11 18:56:05.449955 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:05.449920 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-sm2rw_89de43e0-f41b-4570-af10-b3a3ef063480/global-pull-secret-syncer/0.log" Sep 11 18:56:05.535535 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:05.535499 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-gt5ln_e9c4d02b-cc0d-4812-8fe2-80f5276923ba/konnectivity-agent/0.log" Sep 11 18:56:05.679843 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:05.679809 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-128.ec2.internal_577bf0a7d48f3dd63acd7590896a721f/haproxy/0.log" Sep 11 18:56:10.604702 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:10.604666 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5cc6c58786-qn445_6923f940-4731-40d1-be20-530994fb7394/authorino/0.log" Sep 11 18:56:10.635976 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:10.635941 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-dvxrr_dc27b87c-4bf6-49de-9a7f-a74ee17929f3/manager/0.log" Sep 11 18:56:10.759380 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:10.759348 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-r68m2_51293999-b24d-48cc-8b7c-ad10331ab72e/manager/0.log" Sep 11 18:56:10.885513 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:10.885432 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-79pgz_1e6be196-1d2f-4255-a612-151457495a5f/manager/0.log" Sep 11 18:56:12.637899 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:12.637853 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-s6nf9_3a406a87-d961-4352-b049-eb5476b6f7a5/cluster-monitoring-operator/0.log" Sep 11 18:56:12.676373 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:12.676337 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-csp6n_cc19fa81-e0df-4136-b22c-f04d63132dac/kube-state-metrics/0.log" Sep 11 18:56:12.696643 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:12.696616 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-csp6n_cc19fa81-e0df-4136-b22c-f04d63132dac/kube-rbac-proxy-main/0.log" Sep 11 18:56:12.722081 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:12.722034 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-csp6n_cc19fa81-e0df-4136-b22c-f04d63132dac/kube-rbac-proxy-self/0.log" Sep 11 18:56:12.960031 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:12.960001 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-crss9_b3d754c1-7827-4226-9abf-c2b03605bf3c/node-exporter/0.log" Sep 11 18:56:12.985584 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:12.985557 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-crss9_b3d754c1-7827-4226-9abf-c2b03605bf3c/kube-rbac-proxy/0.log" Sep 11 18:56:13.010530 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:13.010505 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-crss9_b3d754c1-7827-4226-9abf-c2b03605bf3c/init-textfile/0.log" Sep 11 18:56:15.590168 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:15.590127 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kfqdv/perf-node-gather-daemonset-s25xf"] Sep 11 18:56:15.590600 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:15.590584 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2441eb50-2c4d-419c-9d0c-696296aba6ed" containerName="maas-api" Sep 11 18:56:15.590648 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:15.590601 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2441eb50-2c4d-419c-9d0c-696296aba6ed" containerName="maas-api" Sep 11 18:56:15.590688 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:15.590682 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2441eb50-2c4d-419c-9d0c-696296aba6ed" containerName="maas-api" Sep 11 18:56:15.593939 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:15.593921 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-s25xf" Sep 11 18:56:15.596022 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:15.596003 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kfqdv\"/\"default-dockercfg-kn4ch\"" Sep 11 18:56:15.596170 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:15.596138 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kfqdv\"/\"kube-root-ca.crt\"" Sep 11 18:56:15.596796 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:15.596779 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kfqdv\"/\"openshift-service-ca.crt\"" Sep 11 18:56:15.601846 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:15.601817 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kfqdv/perf-node-gather-daemonset-s25xf"] Sep 11 18:56:15.625504 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:15.625466 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgp9p\" (UniqueName: \"kubernetes.io/projected/2da172eb-69f1-48ee-a88a-15ae1ab6dff1-kube-api-access-kgp9p\") pod \"perf-node-gather-daemonset-s25xf\" (UID: \"2da172eb-69f1-48ee-a88a-15ae1ab6dff1\") " pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-s25xf" Sep 11 18:56:15.625689 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:15.625524 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2da172eb-69f1-48ee-a88a-15ae1ab6dff1-podres\") pod \"perf-node-gather-daemonset-s25xf\" (UID: \"2da172eb-69f1-48ee-a88a-15ae1ab6dff1\") " pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-s25xf" Sep 11 18:56:15.625689 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:15.625604 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2da172eb-69f1-48ee-a88a-15ae1ab6dff1-sys\") pod \"perf-node-gather-daemonset-s25xf\" (UID: \"2da172eb-69f1-48ee-a88a-15ae1ab6dff1\") " pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-s25xf" Sep 11 18:56:15.625689 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:15.625643 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2da172eb-69f1-48ee-a88a-15ae1ab6dff1-proc\") pod \"perf-node-gather-daemonset-s25xf\" (UID: \"2da172eb-69f1-48ee-a88a-15ae1ab6dff1\") " pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-s25xf" Sep 11 18:56:15.625689 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:15.625677 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2da172eb-69f1-48ee-a88a-15ae1ab6dff1-lib-modules\") pod \"perf-node-gather-daemonset-s25xf\" (UID: \"2da172eb-69f1-48ee-a88a-15ae1ab6dff1\") " pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-s25xf" Sep 11 18:56:15.726498 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:15.726458 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgp9p\" (UniqueName: \"kubernetes.io/projected/2da172eb-69f1-48ee-a88a-15ae1ab6dff1-kube-api-access-kgp9p\") pod \"perf-node-gather-daemonset-s25xf\" (UID: \"2da172eb-69f1-48ee-a88a-15ae1ab6dff1\") " pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-s25xf" Sep 11 18:56:15.726698 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:15.726514 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2da172eb-69f1-48ee-a88a-15ae1ab6dff1-podres\") pod \"perf-node-gather-daemonset-s25xf\" (UID: \"2da172eb-69f1-48ee-a88a-15ae1ab6dff1\") " pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-s25xf" Sep 11 18:56:15.726698 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:15.726556 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2da172eb-69f1-48ee-a88a-15ae1ab6dff1-sys\") pod \"perf-node-gather-daemonset-s25xf\" (UID: \"2da172eb-69f1-48ee-a88a-15ae1ab6dff1\") " pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-s25xf" Sep 11 18:56:15.726698 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:15.726586 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2da172eb-69f1-48ee-a88a-15ae1ab6dff1-proc\") pod \"perf-node-gather-daemonset-s25xf\" (UID: \"2da172eb-69f1-48ee-a88a-15ae1ab6dff1\") " pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-s25xf" Sep 11 18:56:15.726698 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:15.726616 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2da172eb-69f1-48ee-a88a-15ae1ab6dff1-lib-modules\") pod \"perf-node-gather-daemonset-s25xf\" (UID: \"2da172eb-69f1-48ee-a88a-15ae1ab6dff1\") " pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-s25xf" Sep 11 18:56:15.726698 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:15.726685 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2da172eb-69f1-48ee-a88a-15ae1ab6dff1-proc\") pod \"perf-node-gather-daemonset-s25xf\" (UID: \"2da172eb-69f1-48ee-a88a-15ae1ab6dff1\") " pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-s25xf" Sep 11 18:56:15.726920 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:15.726685 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2da172eb-69f1-48ee-a88a-15ae1ab6dff1-sys\") pod \"perf-node-gather-daemonset-s25xf\" (UID: \"2da172eb-69f1-48ee-a88a-15ae1ab6dff1\") " pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-s25xf" Sep 11 18:56:15.726920 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:15.726723 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2da172eb-69f1-48ee-a88a-15ae1ab6dff1-podres\") pod \"perf-node-gather-daemonset-s25xf\" (UID: \"2da172eb-69f1-48ee-a88a-15ae1ab6dff1\") " pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-s25xf" Sep 11 18:56:15.726920 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:15.726781 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2da172eb-69f1-48ee-a88a-15ae1ab6dff1-lib-modules\") pod \"perf-node-gather-daemonset-s25xf\" (UID: \"2da172eb-69f1-48ee-a88a-15ae1ab6dff1\") " pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-s25xf" Sep 11 18:56:15.735417 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:15.735381 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgp9p\" (UniqueName: \"kubernetes.io/projected/2da172eb-69f1-48ee-a88a-15ae1ab6dff1-kube-api-access-kgp9p\") pod \"perf-node-gather-daemonset-s25xf\" (UID: \"2da172eb-69f1-48ee-a88a-15ae1ab6dff1\") " pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-s25xf" Sep 11 18:56:15.755668 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:15.755643 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dj524_60702ef3-98ef-463b-b877-3a00825353d8/console-operator/1.log" Sep 11 18:56:15.766443 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:15.766406 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dj524_60702ef3-98ef-463b-b877-3a00825353d8/console-operator/2.log" Sep 11 18:56:15.904951 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:15.904918 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-s25xf" Sep 11 18:56:16.182124 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:16.182094 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kfqdv/perf-node-gather-daemonset-s25xf"] Sep 11 18:56:16.183166 ip-10-0-138-128 kubenswrapper[2561]: W0911 18:56:16.183137 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2da172eb_69f1_48ee_a88a_15ae1ab6dff1.slice/crio-bdbb856fe8b28e605cedc5466305242425c9eff38c43414f734e7af24680c8c1 WatchSource:0}: Error finding container bdbb856fe8b28e605cedc5466305242425c9eff38c43414f734e7af24680c8c1: Status 404 returned error can't find the container with id bdbb856fe8b28e605cedc5466305242425c9eff38c43414f734e7af24680c8c1 Sep 11 18:56:16.357886 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:16.357840 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-77bfcd69b7-n58zd_3467da4b-a896-486d-b322-dc59457d7578/console/0.log" Sep 11 18:56:16.954794 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:16.954761 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-s25xf" event={"ID":"2da172eb-69f1-48ee-a88a-15ae1ab6dff1","Type":"ContainerStarted","Data":"30f633cc1221765d30fe57d1c4205140c44e1ea93e66a43aadd17046d34b60e5"} Sep 11 18:56:16.955227 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:16.954810 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-s25xf" event={"ID":"2da172eb-69f1-48ee-a88a-15ae1ab6dff1","Type":"ContainerStarted","Data":"bdbb856fe8b28e605cedc5466305242425c9eff38c43414f734e7af24680c8c1"} Sep 11 18:56:16.955227 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:16.954850 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-s25xf" Sep 11 18:56:16.972973 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:16.972914 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-s25xf" podStartSLOduration=1.972897855 podStartE2EDuration="1.972897855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:56:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:56:16.970296478 +0000 UTC m=+2458.467314244" watchObservedRunningTime="2026-09-11 18:56:16.972897855 +0000 UTC m=+2458.469915634" Sep 11 18:56:17.124112 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:17.124078 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-jfk78_56ef9bd1-1315-4693-93a1-9cf6dc6065a0/volume-data-source-validator/0.log" Sep 11 18:56:18.295022 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:18.294991 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-c6nk8_71d0b928-d82f-4a8d-844c-36613596b07b/dns/0.log" Sep 11 18:56:18.315942 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:18.315914 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-c6nk8_71d0b928-d82f-4a8d-844c-36613596b07b/kube-rbac-proxy/0.log" Sep 11 18:56:18.492339 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:18.492308 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xh28n_7f7d5659-22fb-43f1-8970-47837d126795/dns-node-resolver/0.log" Sep 11 18:56:19.055450 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:19.055412 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6cd6c4f7f7-vh7q7_c6f21f03-eca4-45e5-80c1-63dee730248d/registry/0.log" Sep 11 18:56:19.082367 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:19.082331 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-k7qjg_6547cc6d-cb31-474a-8065-7fd47766fe02/node-ca/0.log" Sep 11 18:56:20.107449 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:20.107413 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7_61673abc-9544-4cf5-aad1-4714f9aacabc/istio-proxy/0.log" Sep 11 18:56:20.601154 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:20.601120 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-67fd44999f-t577s_7c271705-ae8e-4715-bb87-d176e6e0ed07/kube-auth-proxy/0.log" Sep 11 18:56:20.633450 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:20.633413 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-cbvsl_09816140-ebbc-4f2d-9e1a-a11e86d773a4/istio-proxy/0.log" Sep 11 18:56:20.638539 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:20.638509 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-cbvsl_09816140-ebbc-4f2d-9e1a-a11e86d773a4/istio-proxy/1.log" Sep 11 18:56:20.678570 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:20.678533 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-ldlb4_cf29bed6-97b3-4988-aa11-ffebe83a653a/payload-pre-processing/0.log" Sep 11 18:56:21.714256 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:21.714224 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-x6rnn_2c3feaeb-94a3-40e2-b158-976547caf818/serve-healthcheck-canary/0.log" Sep 11 18:56:22.211829 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:22.211796 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-spcnr_97808c4d-b62a-4ca1-b686-5e27a9aa79a0/insights-operator/0.log" Sep 11 18:56:22.212513 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:22.212478 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-spcnr_97808c4d-b62a-4ca1-b686-5e27a9aa79a0/insights-operator/1.log" Sep 11 18:56:22.319743 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:22.319713 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-c2vf2_b8a85f9a-991b-4751-925e-4e4a8a400a90/kube-rbac-proxy/0.log" Sep 11 18:56:22.342313 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:22.342286 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-c2vf2_b8a85f9a-991b-4751-925e-4e4a8a400a90/exporter/0.log" Sep 11 18:56:22.364339 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:22.364308 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-c2vf2_b8a85f9a-991b-4751-925e-4e4a8a400a90/extractor/0.log" Sep 11 18:56:22.970525 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:22.970495 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-s25xf" Sep 11 18:56:24.930001 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:24.929969 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-rfphm_1329cb52-610f-4b0f-8b8b-102288987992/manager/0.log" Sep 11 18:56:25.438442 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:25.438400 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-tqgww_052c1dcf-bad7-47b8-b5da-8a1ef2459f46/manager/0.log" Sep 11 18:56:25.515264 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:25.515239 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-l6vbm_b9832d55-a490-4d4e-8b98-3b4f676f42e9/server/0.log" Sep 11 18:56:25.669393 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:25.669360 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-mnwhs_bf80fc01-7343-4f52-a954-dcf935983c7c/manager/0.log" Sep 11 18:56:25.892345 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:25.892313 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76cff9b6b6-rg86p_a5a6a746-6467-4629-b647-250c51a1e91f/manager/0.log" Sep 11 18:56:25.913873 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:25.913839 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76cff9b6b6-rg86p_a5a6a746-6467-4629-b647-250c51a1e91f/copy-manifests/0.log" Sep 11 18:56:26.617503 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:26.617469 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7_61673abc-9544-4cf5-aad1-4714f9aacabc/istio-proxy/0.log" Sep 11 18:56:27.115840 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:27.115810 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-67fd44999f-t577s_7c271705-ae8e-4715-bb87-d176e6e0ed07/kube-auth-proxy/0.log" Sep 11 18:56:27.140364 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:27.140318 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-cbvsl_09816140-ebbc-4f2d-9e1a-a11e86d773a4/istio-proxy/0.log" Sep 11 18:56:27.145940 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:27.145916 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-cbvsl_09816140-ebbc-4f2d-9e1a-a11e86d773a4/istio-proxy/1.log" Sep 11 18:56:27.176563 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:27.176535 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-ldlb4_cf29bed6-97b3-4988-aa11-ffebe83a653a/payload-pre-processing/0.log" Sep 11 18:56:27.214690 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:27.214660 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-g4bsz_850b2548-8f04-42ef-ab35-bc6a01bc0053/openshift-lws-operator/0.log" Sep 11 18:56:28.739797 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:28.739761 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-vtsv7_61673abc-9544-4cf5-aad1-4714f9aacabc/istio-proxy/0.log" Sep 11 18:56:29.238589 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:29.238556 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-67fd44999f-t577s_7c271705-ae8e-4715-bb87-d176e6e0ed07/kube-auth-proxy/0.log" Sep 11 18:56:29.263139 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:29.263093 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-cbvsl_09816140-ebbc-4f2d-9e1a-a11e86d773a4/istio-proxy/0.log" Sep 11 18:56:29.268448 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:29.268416 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-cbvsl_09816140-ebbc-4f2d-9e1a-a11e86d773a4/istio-proxy/1.log" Sep 11 18:56:29.304364 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:29.304340 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-ldlb4_cf29bed6-97b3-4988-aa11-ffebe83a653a/payload-pre-processing/0.log" Sep 11 18:56:32.784733 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:32.784701 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-9pjz5_73a83e98-5fd7-43a6-85cc-ffd8d36057cf/kube-storage-version-migrator-operator/1.log" Sep 11 18:56:32.786509 ip-10-0-138-128 kubenswrapper[2561]: I0911 18:56:32.786482 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-9pjz5_73a83e98-5fd7-43a6-85cc-ffd8d36057cf/kube-storage-version-migrator-operator/0.log"